Multi-part Uploads
Sentera's GraphQL API supports uploading a file to Sentera's cloud storage using a multi-part upload. This approach is recommended for larger files (10MB or greater), and the only way to upload files larger than 5GB.
Uploading a file with this approach and then associating it with a FieldAgent resource requires 5 steps, which are illustrated in the diagram below.
sequenceDiagram
API Client->>+GraphQL API: Create multi-part file upload
GraphQL API->>+AWS S3: Create multi-part file upload
AWS S3-->-GraphQL API: upload_id
GraphQL API-->-API Client: upload_id, s3_key, file_id
loop over the number of parts as determined by the API Client
API Client->>+GraphQL API: 2. Prepare a part (upload ID, s3 key, part number)
GraphQL API-->-API Client: upload URL
API Client->>+AWS S3: 3. Upload the part (upload URL, part)
AWS S3-->-API Client: entity tag
end
alt File uploaded to S3 successfully
API Client->>+GraphQL API: 4. Complete the multi-part file upload (upload id, s3 key, part numbers, entity tags)
GraphQL API->>+AWS S3: Complete the multi-part file upload (upload id, s3 key, entity tags)
AWS S3-->-GraphQL API: OK
GraphQL API-->-API Client: OK
else File upload failed or canceled
API Client->>+GraphQL API: 4. Abort the multi-part file upload (upload ID, s3 key)
GraphQL API->>+AWS S3: Abort the multi-part file upload (upload ID, s3 key)
AWS S3-->-GraphQL API: OK
GraphQL API-->-API Client: OK
end
If you prefer to dive into some code and get going, there are working code examples that demonstrate the multi-part file upload workflow available at GitHub.
Step 1 - Create a Multipart File Upload
The first thing you need to do is to tell Sentera's platform that you want to upload a file using a multi-part upload, by creating a multi-part file upload.
To do this, use the create_multipart_file_upload GraphQL mutation.
mutation CreateMultipartFileUpload {
create_multipart_file_upload(
byte_size: 27480452
content_type: "image/tiff"
filename: "test.tif"
file_upload_owner: {
parent_sentera_id: "7hy7bwv_CO_cyscAcme_CV_deve_6e3c36d3f_220308_135409"
owner_type: MOSAIC
}
) {
file_id
owner_sentera_id
s3_key
upload_id
}
}
You will use the values returned by this API call in steps 2, 3, 5, and 5.
Steps 2 and 3 - Upload the File in Parts
The next steps in this workflow are to upload the file in parts to AWS S3.
The response from the create_multipart_file_upload
mutation in step 1 contains an upload ID and S3 key that you will use to upload each part of the file.
How you upload the file in parts to AWS S3 is up to you.
AWS provides a multi-part SDK
that you can use. Or, you can make individual PUT requests with the URLs obtained
from requests to the prepare_multipart_file_upload_part or prepare_multipart_file_upload_parts
mutations to upload the parts to AWS S3.
mutation PrepareMultipartFileUploadPart {
prepare_multipart_file_upload_part(
part_number: 1
s3_key: "cyscAcmeOrg/fnnb6nz_AS_cyscAcme_CV_deve_7bbde271d_230313_103846/sre9bzg_CO_cyscAcme_CV_deve_7bbde271d_230313_103852/FeatureSets/z5rhe13_FS_cyscAcme_CV_deve_e0a089e79_230313_172119/test.geojson"
upload_id: "kU_5uPVH_pGOud5ScQlCQs0dX2Q7gu.Pn80EB7M1EiYWhV2SthcHdGFt2ov2Awe2or1g0rUcRMU6.MrxPn4Pw4cS2HIcTnj6AJOoZCmaBZZZ6xv8r8jVCtVdZpUxYG17"
) {
part_number
url
}
}
mutation PrepareMultipartFileUploadParts {
prepare_multipart_file_upload_parts(
part_numbers: [1, 2, 3]
s3_key: "cyscAcmeOrg/fnnb6nz_AS_cyscAcme_CV_deve_7bbde271d_230313_103846/sre9bzg_CO_cyscAcme_CV_deve_7bbde271d_230313_103852/FeatureSets/z5rhe13_FS_cyscAcme_CV_deve_e0a089e79_230313_172119/test.geojson"
upload_id: "kU_5uPVH_pGOud5ScQlCQs0dX2Q7gu.Pn80EB7M1EiYWhV2SthcHdGFt2ov2Awe2or1g0rUcRMU6.MrxPn4Pw4cS2HIcTnj6AJOoZCmaBZZZ6xv8r8jVCtVdZpUxYG17"
) {
part_number
url
}
}
Step 4 Complete the Multipart File Upload
Once you have completed uploading the parts to AWS S3, you need to indicate that the upload is complete
by using the complete_multipart_file_upload mutation.
mutation CompleteMultipartFileUpload {
complete_multipart_file_upload(
parts: [
{
part_number: 1
etag: "e8a621777ce299758dadc46b3bf3b176"
}
{
part_number: 2
etag: "4da2f9e7c26d4017b0a9556dfd6f08c6"
}
{
part_number: 3
etag: "7bk2f9e7c26d4017b0a9556dfd6f02p3"
}
]
s3_key: "cyscAcmeOrg/fnnb6nz_AS_cyscAcme_CV_deve_7bbde271d_230313_103846/sre9bzg_CO_cyscAcme_CV_deve_7bbde271d_230313_103852/FeatureSets/z5rhe13_FS_cyscAcme_CV_deve_e0a089e79_230313_172119/test.geojson"
upload_id: "kU_5uPVH_pGOud5ScQlCQs0dX2Q7gu.Pn80EB7M1EiYWhV2SthcHdGFt2ov2Awe2or1g0rUcRMU6.MrxPn4Pw4cS2HIcTnj6AJOoZCmaBZZZ6xv8r8jVCtVdZpUxYG17"
)
}
Step 4 Alternative - Abort the Multipart File Upload
If you want to cancel the upload or the upload cannot be completed successfully for whatever reason,
use the abort_multipart_file_upload mutation. Note that any partial uploads that have not
been explicitly aborted will be automatically removed from AWS S3 after 30 days.
mutation AbortMultipartFileUpload {
abort_multipart_file_upload(
s3_key: "cyscAcmeOrg/fnnb6nz_AS_cyscAcme_CV_deve_7bbde271d_230313_103846/sre9bzg_CO_cyscAcme_CV_deve_7bbde271d_230313_103852/FeatureSets/z5rhe13_FS_cyscAcme_CV_deve_e0a089e79_230313_172119/test.geojson"
upload_id: "kU_5uPVH_pGOud5ScQlCQs0dX2Q7gu.Pn80EB7M1EiYWhV2SthcHdGFt2ov2Awe2or1g0rUcRMU6.MrxPn4Pw4cS2HIcTnj6AJOoZCmaBZZZ6xv8r8jVCtVdZpUxYG17"
)
}
Step 4 - Associate the File with a FieldAgent Resource
The final step in the workflow is to use the file upload's id that you obtained in step 1
with a GraphQL mutation. This ID identifies the file you uploaded to AWS S3, and will be used by the
Sentera platform to associate the file with a resource in your organization, such as a field, survey,
feature set, etc.
For example, you could use the import_mosaic
mutation to import a GeoTIFF file that you uploaded using a multi-part upload as a mosaic in FieldAgent.