Set_s3_config() only supports the default region

[AbuQutaita]

It appears that set_s3_config() expects the S3 bucket doesn't exist and will create it. It also doesn't allow specifying the AWS region to create the new bucket. Ideally one or both of the following would be supported:

1. Support using an S3 bucket if it already exists.
2. Take the AWS region as a parameter so the bucket can be created in a non-default region.

Thanks

[Brian]

Currently only the default us-east-1 region is supported. But we could probably add support for other regions in the next release.

[Brian]

Version 2.3.0 introduces an option to specify the region:

QuantRocket 2.3.0 Release Notes

QuantRocket 2.3.0 is now available. This release includes improvements for teams and financial advisors, and other enhancements and bug fixes. All version 2 users are encouraged to update to the latest version. See how to update. Highlights improvements for teams, including a dedicated documentation section, support for linked license keys, and support for read-only S3 credentials. See the teams documentation new dedicated documentation section for financial advisors, and support for placing…

[AbuQutaita]

We ended up creating our QR environment in AWS and decided to bring it up in us-west-2. However, this setting doesn't appear to be working. It gets stucks pushing the master DB:

2021-09-21 16:52:51 quantrocket.db: ERROR Traceback (most recent call last):
2021-09-21 16:52:51 quantrocket.db: ERROR   File "sym://qrocket_log_py", line 34, in wrapped
2021-09-21 16:52:51 quantrocket.db: ERROR   File "sym://qrocket_mule_py", line 24, in mule_s3_push
2021-09-21 16:52:51 quantrocket.db: ERROR   File "sym://qrocket_db_s3_py", line 192, in push
2021-09-21 16:52:51 quantrocket.db: ERROR   File "sym://qrocket_db_s3_py", line 226, in _push_sqlite
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/s3/key.py", line 1375, in set_contents_from_filename
2021-09-21 16:52:51 quantrocket.db: ERROR     return self.set_contents_from_file(fp, headers, replace, cb,
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/s3/key.py", line 1307, in set_contents_from_file
2021-09-21 16:52:51 quantrocket.db: ERROR     self.send_file(fp, headers=headers, cb=cb, num_cb=num_cb,
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/s3/key.py", line 760, in send_file
2021-09-21 16:52:51 quantrocket.db: ERROR     self._send_file_internal(fp, headers=headers, cb=cb, num_cb=num_cb,
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/s3/key.py", line 957, in _send_file_internal
2021-09-21 16:52:51 quantrocket.db: ERROR     resp = self.bucket.connection.make_request(
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/s3/connection.py", line 667, in make_request
2021-09-21 16:52:51 quantrocket.db: ERROR     return super(S3Connection, self).make_request(
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/connection.py", line 1070, in make_request
2021-09-21 16:52:51 quantrocket.db: ERROR     return self._mexe(http_request, sender, override_num_retries,
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/connection.py", line 1030, in _mexe
2021-09-21 16:52:51 quantrocket.db: ERROR     raise ex
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/connection.py", line 939, in _mexe
2021-09-21 16:52:51 quantrocket.db: ERROR     response = sender(connection, request.method, request.path,
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/site-packages/boto/s3/key.py", line 856, in sender
2021-09-21 16:52:51 quantrocket.db: ERROR     http_conn.send(chunk)
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/http/client.py", line 971, in send
2021-09-21 16:52:51 quantrocket.db: ERROR     self.sock.sendall(data)
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/ssl.py", line 1204, in sendall
2021-09-21 16:52:51 quantrocket.db: ERROR     v = self.send(byte_view[count:])
2021-09-21 16:52:51 quantrocket.db: ERROR   File "/opt/conda/lib/python3.8/ssl.py", line 1173, in send
2021-09-21 16:52:51 quantrocket.db: ERROR     return self._sslobj.write(data)
2021-09-21 16:52:51 quantrocket.db: ERROR BrokenPipeError: [Errno 32] Broken pipe
2021-09-21 16:52:51 quantrocket.db: ERROR

What's curious is DBs before this get pushed correctly. I've confirmed the IAM policies for the buckets are the same, and also confirmed I'm using the --region 'us-west-2' correctly for s3config.

Thanks

[Brian]

Broken pipe often indicates some kind of network disconnection for whatever reason and I would normally expect that to be transient. Does it repeat consistently for the same database? I’m doubtful this has to do with QuantRocket as it's an unhandled exception in the boto library.

[AbuQutaita]

Yep, repros 100% of the time, always getting stuck on master. Same issue for other buckets in the same region (us-west-2). Starts working again as soon as I set the s3 config to a bucket in us-east-1.

[Brian]

It sounds like this can happen with certain files in non-standard regions using boto. The fact that it randomly affects specific files is probably why it happens to you but not in our test suite. We’ll make some adjustments to try to make boto happier.

[AbuQutaita]

Sounds good. Feel free to send a patch my way and I’ll be happy to confirm if it fixes the issue since it’s somewhat transient.

[Brian]

This is now patched in quantrocket/db:latest. Please edit your docker-compose.yml and pull this image to test the fix.

[AbuQutaita]

I confirmed that this fixed the issue; the master db was successfully pushed this time.

I'm curious, what was the nature of the fix?