File Stores

File Stores are a filesystem that are scoped to a Project. All workflows within the same Project have access to the File Stores.

You can interact with these files through the Pipedream Dashboard or programmatically through your Project's workflows.

Unlike files stored within a workflow's /tmp directory which are subject to deletion between executions, File Stores are separate cloud storage. Files within a File Store can be long term storage accessible by your workflows.

Managing File Stores from the Dashboard

You can access a File Store by opening the Project and selecting the File Store on the left hand navigation menu.

Uploading files to the File Store

To upload a file, select New then select File:

Then in the new pop-up, you can either drag and drop or browser your computer to stage a file for uploading:

Now that the file(s) are staged for uploaded. Click Upload to upload them:

Finally, click Done to close the upload pop-up:

You should now see your file is uploaded and available for use within your Project:

Deleting files from the File Store

You can delete individual files from a File Store by clicking the three dot menu on the far right of the file and selecting Delete.

After confirming that you want to delete the file, it will be permanently deleted.

Managing File Stores from Workflows

Files uploaded to a File Store are accessible by workflows within that same project.

You can access these files programmatically using the $.files helper within Node.js code steps.

Listing files in the File Store

The $.files.dir() method allows you to list files and directories within the Project's File Store. By default it will list the files at the root directory.

Here's an example of how to iterate over the files in the root directory and open them as File instances:

export default defineComponent({
  async run({ steps, $ }) {
    // list all contents of the root File Stores directory in this project
    const dirs = $.files.dir();
    let files = [];
 
    for await(const dir of dirs) {
      // if this is a file, let's open it
      if(dir.isFile()) {
        files.push(dir.path)
      }
    }
 
    return files
  },
})

Opening files

To interact with a file uploaded to the File Store, you'll first need to open it.

Given there's a file in the File Store called example.png, you can open it using the $.files.open() method:

export default defineComponent({
  async run({ steps, $ }) {
    // Open the file by it's path in the File Store
    const file = $.files.open('example.png')
    // Log the S3 url to access the file publicly
    return await file.toUrl()
  },
})

Once the file has been opened, you can read, write, delete the file and more.

Uploading files to File Stores

You can upload files using Node.js code in your workflows, either from URLs, from the /tmp directory in your workflows or directly from streams for high memory efficency.

Uploading files from URLs

File.fromUrl() can upload a file from a public URL to the File Store.

First open a new file at a specific path in the File Store, and then pass a URL to the fromUrl method on that new file:

export default defineComponent({
  async run({ steps, $ }) {
    // Upload a file to the File Store by a URL
    const file = await $.files.open('pipedream.png').fromUrl('https://res.cloudinary.com/pipedreamin/image/upload/t_logo48x48/v1597038956/docs/HzP2Yhq8_400x400_1_sqhs70.jpg')
 
    // display the uploaded file's URL from the File Store:
    console.log(await file.toUrl())
  },
})

Uploading files from the workflow's /tmp directory

File.fromFile() can upload a file stored within the workflow's /tmp directory to the File Store.

First open a new file at a specific path in the File Store, and then pass a URL to the fromFile method on that new file:

export default defineComponent({
  async run({ steps, $ }) {
    // Upload a file to the File Store from the local /tmp/ directory
    const file = await $.files.open('recording.mp3').fromFile('/tmp/recording.mp3')
 
    // Display the URL to the File Store hosted file
    console.log(await file.toUrl())
  },
})

Uploading files using streams

File Stores also support streaming to write large files. File.createWriteStream() creates a write stream for the file to upload to. Then you can pair this stream with a download stream from another remote location:

import { pipeline } from 'stream/promises';
import got from 'got'
 
export default defineComponent({
  async run({ steps, $ }) {
    const writeStream = await $.files.open('logo.png').createWriteStream()
 
    const readStream = got.stream('https://pdrm.co/logo')
 
    await pipeline(readStream, writeStream);
  },
})

Additionally, you can pass a ReadableStream instance directly to a File instance:

import got from 'got'
 
export default defineComponent({
  async run({ steps, $ }) {
    // Start a new read stream
    const readStream = got.stream('https://pdrm.co/logo')
 
    // Populate the file's content from the read stream
    await $.files.open("logo.png").fromReadableStream(readStream)
  },
})

Downloading files

File Stores live in cloud storage by default, but files can be downloaded to your workflows individually.

Downloading files to the workflow's /tmp directory

First open a new file at a specific path in the File Store, and then call the toFile() method to download the file to the given path:

import fs from 'fs';
 
export default defineComponent({
  async run({ steps, $ }) {
    // Download a file from the File Store to the local /tmp/ directory
    const file = await $.files.open('recording.mp3').toFile('/tmp/README.md')
 
    // read the file version of the file stored in /tmp
    return (await fs.promises.readFile('/tmp/README.md')).toString()
  },
})

Passing files between steps

Files can be passed between steps. Pipedream will automatically serialize the file as a JSON description of the file. Then when you access the file as a step export as a prop in a Node.js code step, then you can interact with the File instance directly.

For example, if you have a file stored at the path logo.png within your File Store, then within a Node.js code step you can open it:

// "open_file" Node.js code step
export default defineComponent({
  async run({ steps, $ }) {
    // Return data to use it in future steps
    const file = $.files.open('logo.png')
 
    return file
  },
})

Then in a downstream code step, you can use it via the steps path:

// "get_file_url" Node.js code step
export default defineComponent({
  async run({ steps, $ }) {
    // steps.open_file.$return_value is automatically parsed back into a File instance:
    return await steps.open_file.$return_value.toUrl()
  },
})

Handling lists of files

One limitation of the automatic parsing of files between steps is that it currently doesn't automatically handle lists of files between steps.

For example, if you have a step that returns an array of File instances:

// "open_files" Node.js code step
export default defineComponent({
  async run({ steps, $ }) {
    // Return data to use it in future steps
    const file1 = $.files.open('vue-logo.svg')
    const file2 = $.files.open('react-logo.svg')
 
    return [file1, file]
  },
})

Then you'll need to use $.files.openDescriptor to parse the JSON definition of the files back into File instances:

// "parse_files" Node.js code step
export default defineComponent({
  async run({ steps, $ }) {
    const files = steps.open_files.$return_value.map(object => $.files.openDescriptor(object))
 
    // log the URL to the first File
    console.log(await files[0].toUrl());
  },
})

Deleting files

You can call delete() on the file to delete it from the File Store.

export default defineComponent({
  async run({ steps, $ }) {
    // Open the file and delete it
    const file = await $.files.open('example.png').delete()
    console.log('File deleted.')
  },
})

Frequently Asked Questions

Are there size limits for files within File Stores?

At this time no, but File Stores are in preview and are subject to change.

Are helpers available for Python to download, upload and manage files?

At this time no, only Node.js includes a helper to interact with the File Store programmatically within workflows.

Are File Stores generally available?

At this time File Stores are only available to Advanced plan and above subscribed workspaces. You can change your plan within the pricing page.