databricks/node-bitdotio

databricks/node-bitdotio

Description: Node SDK for bit.io

Language: TypeScript

License: MIT

Stars: 3

Forks: 0

Open issues: 1

Created: 2022-11-23T17:32:50Z

Pushed: 2023-07-20T02:46:50Z

Default branch: main

Fork: no

Archived: yes

README:

bit.io Node.js SDK

Node.js SDK for connecting to bit.io databases and interacting with the bit.io developer API.

Installation

npm install @bitdotioinc/node-bitdotio

Versioning

node-bitdotio uses semantic versioning.

Usage

The node-bitdotio package consists of a bitdotio SDK object which provides helpful utilities for creating and managing pre-configured connections to bit.io databases via `node-postgres`, as well as easy methods for accessing the functionality exposed by the bit.io developer API.

Once you have node-bitdotio installed all you need is your API key to start working with bit.io.

You can get your API key by logging into bit.io and opening the "Connect" tab on a database page.

Example usage

See API reference at the bottom of this document for a full list of methods provided by the SDK.

const { createReadStream } = require('fs');
const bitdotio = require('@bitdotioinc/node-bitdotio');

const BITIO_KEY = process.env.BITIO_KEY;

// Instantiate the bitdotio SDK
const b = bitdotio(BITIO_KEY);

// Create a database
b.createDatabase({ name: "demo-db" }).then(async (dbMetadata) => {
const dbName = dbMetadata.name;

// Create a connection pool for our newly created database
const pool = b.getPool(dbName);

// Connect to the database using the pool and insert some data
await pool.connect().then(async (client) => {
await client.query("CREATE TABLE test (foo integer, bar text)");
const data = [[1, "one"], [2, "two"], [3, "three"]];
for (const row of data) {
await client.query("INSERT INTO test (foo, bar) VALUES ($1, $2)", row);
}
// Make sure to return all clients acquired from a connection pool
client.release();
});

// Run a query against our new table using the API
await b.query(dbName, "SELECT * FROM test", "objects").then((results) => {
console.table(results.data);
});

const sleep = (ms) => new Promise((resolve) => { setTimeout(resolve, ms); });

// Start a file upload job and wait for it to complete
const file = createReadStream('./cities.csv');
let jobStatus = await b.createImportJob(dbName, { tableName: "cities", type: "file", file });
while (true) {
jobStatus = await b.getImportJob(jobStatus.id);
if (jobStatus.state === "DONE") {
break;
}
if (jobStatus.state === "FAILED") {
const { errorId, errorType, errorDetails } = jobStatus;
throw new Error(
`Import job failed. Error ID: ${errorId}, Error Type: ${errorType}, Error Details: ${errorDetails}`
);
}

// Sleep a bit to not overload the API
await sleep(200);
}
});

Making queries

The node-bitdotio SDK offers two different methods for executing queries against a bit.io database: 1. Connecting directly via the Postgres protocol using node-postgres 2. Querying over HTTP using the bit.io developer API

The tradeoffs between the two are worth some discussion here. In short, direct connections are preferable in the majority of cases, and the query method should be used sparingly, or wherever it is not possible to use a database driver or a raw TCP socket.

Connecting directly

The bitdotio SDK object provides the methods getPool and getClient, which return instances of `pg.Pool` and `pg.Client`, respectively. Both of these are used to establish direct connections to your bit.io database. The former makes use of a connection pool, which manages multiple connections and ensures they are kept alive, and is more suitable to long-running applications; the latter directly establishes a single, unmanaged connection to you database, and is more suitable for short scripts and the like. More information on connecting can be found in the node-postgres documentation here.

It is preferable to use direct connections in the majority of cases since they have superior performance and enable more features compared to the query method. In particular we strongly recommend using direct connections in programs that are long-running, require transaction management, or transfer large quantities of data.

The query method

The query runs queries via the bit.io HTTP API, and therefore will have worse performance and feature support than direct connections. Importantly, each query run via the query method is run in a single transaction. The query method is recommended in situations where installing a database driver is undesirable or impossible, when queries are being run very infrequently, or in very short-lived contexts such as one-off scripts or serverless backends.

Connection pooling and management [important]

The recommended way to obtain a direct connection to a bit.io database is via a connection pool. In order to support scaling to zero, bit.io automatically closes idle connections after a period of time, and puts databases into a dormant state when there are no live connections. If you are designing a long-running application, you should make sure that your database access pattern is resilient to connection closures and database shutdowns. The best way to do this is via a connection pool. Acquiring connections from a connection pool allows connection re-use, and handles reconnects in the event that a connection is dropped. You can instantiate a connection pool for a given database using the bitdotio SDK object's getPool method. See the `pg.Pool` documentation for information on how to connect from a pool. If using connectin pools, it is also recommended to gracefully tear down the bitdotio SDK object when your application terminates using the end method.

There may be situations in which a self-managed, unpooled connection is needed. For example, if the client needs to persist state onto the connection's database session using the SET command. For such situations, the SDK object provides the getClient method.

Data imports and exports

The bitdotio SDK object provides helper methods to facilitate importing and exporting data from your bit.io…