Topics

Tribe Support

Tribe Support's Topics

Scott Crawford

Scott Crawford

July 09 2019

Connecting from Amazon EC2 to S3 not working

I've made several attempts starting from scratch to get S3 working without functional results.  Within AWS's Security Credentials center I've set up a IAM user and assigned AWS's default 'AmazonS3FullAccess' permissions policy.  For the user I then generate the Access Keys and copy them to Anahita's plugin, along with the S3 bucket name and folder - both of which are setup in the S3 storage to allow public access (I've put a few image files in these folders and confirmed I can access them publicly).  But, when I go to change the Site Admin's avatar nothing is uploaded / no folder for the node is created on S3, although the "broken" image on Anahita's front end is pointing to an S3 location so the plugin should be active.

Comparing my EC2 install to getanahita.com, I see the bucket naming convention in the URL's are different:

  • My getanahita.com avatar (working) = s3.amazonaws.com/[bucketname]/assets/public
  • My EC2 Anahita avatar (not working) = [bucketname].s3.amazonaws.com/assets/public

So I've put a test file in a different folder in my bucket and opened it publicly.  It's URL includes the AWS region:
  • Test file location (working) = [bucketname].s3.us-east-2.amazonaws.com/media/logo.png

I can remove the AWS region and the file is still viewable:

  • Test file location (working) = [bucketname].s3.amazonaws.com/media/logo.png
But, when I attempt to load the test file using the getanahita.com URL convention the file will not load:
  • Test file location (NOT working) = s3.amazonaws.com/[bucketname]./media/logo.png
Could this be something in the plugin that's broken?Since I've overlooked a few php modules and configurations in setting up on EC2, my other question would be whether something in my php setup may be preventing Anahita from connecting to S3?

#EC2 #AWS
I think AWS S3 has changed its path conventions and getanahita.com is using the legacy paths, and our bucket is from years ago. Perhaps we need to update the S3 plugin and make it work in both legacy and current mode.
Although the PHP AWS S3 library perhaps needs to be upgraded too.
Scott Crawford
Scott Crawford
July 10 2019 Permalink
Is the PHP AWS S3 library you're referring to something I'd need to install separately / i.e., is it a PHP module? Or included within Anahita's plugin?

I see that the plugin considers whether SSL is used in determining whether to juxtapose the bucket name's position in the URL. I haven't yet set up SSL, so it appears the convention should be [bucketname].s3.amazonaws.com, in which case the URL would match in my install and be the same as the test files I can access publicly from S3.
I would set up the SSL first before attempting to update the plugin.

The AWS PHP library is this file https://github.com/anahitasocial/anahita/blob/297d08977a9269ff9a29a879aadc95f2f4865e5c/src/plugins/storage/s3lib.php

, but quite likely they have a new version out.
Unknown Person liked this
Scott Crawford
Scott Crawford
July 11 2019 Permalink
That makes sense, very much appreciated.
Rastin Mehr liked this
Scott Crawford
Scott Crawford
July 16 2019 Permalink
Revisiting this issue. On my EC2 instance, the DNS records now correctly point to the root anahita www directory, and SSL is enabled and confirmed permanently redirecting to HTTPS. Still, though, the S3 connection is not functioning.

Aside from any AWS Security-related paths to look into, would there be anything relating to Anahita that I should also investigate? Understanding the s3lib might need to be updated, would there be specific php modules required? Or perhaps file permissions that might be worth looking into? On the s3lib file itself, it seems the link provided in the header is no longer active.

Worst case, if we start to use this as a live project without S3 enabled, is it possible to activate S3 storage in Anahita after using for a period of time without using S3?
So which part isn't working? Is it the file upload from EC2 to an S3 bucket? Or files upload fine, but the s3 Anahita plugin isn't constructing the paths correctly?
Scott Crawford
Scott Crawford
July 16 2019 Permalink
For our use-case it's only the profile images (avatars and covers) that will be affected for now. We're not yet installing photos or any file-related Anahita apps. So it's Anahita on EC2 that I can't seem to connect to S3 despite having set up an "IAM" user with full S3 privileges and keys recorded in Anahita's S3 plugin. When I upload an avatar, no node folders are created on S3, and Anahita's front end displays a "broken image". During setup I did manually created the "assets" folder in the S3 bucket and made it public.

I've only had a little time today to dig further, but I am finding indication that S3 now differentiates between website endpoints and REST API requests. At first glance I have the impression that the plugin s3lib.php file might just need some slight modifications to accommodate the changes, but again I haven't had time to test.

Here's the references I've come across so far:

StackOverflow (see the green-checked answer):
https://stackoverflow.com/questions/26604977/url-for-public-amazon-s3-bucket

Amazon AWS documentation:
https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteEndpoints.html
Avatar and cover use the same underlying mechanism for file upload as the photos app. I was going to mention the IAM next. Make sure the AWS permissions are set correctly for an app within EC2 to be able to upload to a bucket. If that didn't work, go ahead with updating the s3 plugin. Install the most recent s3lib and make any changes to the plugin if necessary. I'd appreciate it if you do a PR please when you get it working.
Unknown Person liked this
Scott Crawford
Scott Crawford
July 18 2019 Permalink
I'm starting to get into the plugin now. I have found on GitHub an S3 class for php that appears to have been based off the existing Anahita plugin library. Running a diff comparison though it appears inclusive of a number of revisions:

https://github.com/tpyo/amazon-s3-php-class/blob/master/S3.php

In testing my setup:

1) I've been able to access S3 from the EC2 server's command line, so that connection appears functional.

2) I re-created a new IAM user and transferred the new credentials into Anahita but that did not resolve the issue. I have also attempted with the root keys, also without success.

3) I've revised the s3.php file's URL construction. The issue still persists, although the URL where it looks for the (not) stored avatar does match the pattern for which I can successfully access test files stored on the S3 instance. So, it seems juxtaposing the URL alone does not resolve the connection issue.

And as of this writing I have substituted the GitHub class for the s3lib.php file and it does not resolve the issue, even with the updated URL construction in the s3.php file. Continuing to dig further...
Rastin Mehr liked this
Look into IAM and whether EC2 can upload any files to the S3 bucket using any PHP scripts.
Scott Crawford
Scott Crawford
July 22 2019 Permalink
This, at least in part, appears it may be related to persistent permission conflicts that I had encountered previously. I must have missed a step, but can't figure out what it would have been.

When an EC2 instance is launched, the default user is "ec2-user", and that user is included in a group called "ec2-user".

After installing Apache, I added the ec2-user to the "www" group, and proceeded to install Anahita. Accordingly, I thought everything would have installed under the "www" group.

It appears though, that Anahita was instead installed under the "ec2-user" group, preventing writing to Anahita's www directory (including configurations.php, the /cache directory, and creating the /assets directory).

So, when installing Anahita how would I ensure it is being installed under the "www" group? Or would it be required to chown everything recursively after the fact?
Yes, try changing it manually.
Scott Crawford
Scott Crawford
July 23 2019 Permalink
So, it seems the 3rd party reference I was following was not matched to my configuration. Their instructions were to add the ec2-user to the "www" group, whereas the solution that works is to change owner and group of the /var/www directory to the "apache" user and the "apache" group.

This resolves all of the writing conflicts encountered, and gets Anahita running smoothly using local storage for uploads. Next task is to tackle the S3 plugin, for which I'll open a separate topic.

I should be able to post an article on getting Anahita running on EC2, just need to clean up the write-up. Will aim to post later today.
Rastin Mehr liked this
I can't wait to get proper release engineering tools in place for Anahita using Docker and Kubernetes. That's the right way to go about it.
Unknown Person liked this

Powered by Anahita