Archive

Posts Tagged ‘amazon’

Amazon EC2 t1.micro swizz !

January 6, 2011 5 comments

Just got my bill from Amazon for the 2 instances Im running and find Ive been charged for 728 hours on one of them – I thought this was supposed to be free for a year ! Reading again the small print (ugh) it seems you are entitled to 750 hours free, but it doesn’t explicitely say per instance. So – it seems its per account and you can run as many instances as you like and use a total of 750 hours across them in total before you get charged. Then again, I suppose thats reasonable enough – Amazon wouldn’t want to have every SME in the world running in the cloud for free, for a year when you could be getting cash from them, would you ? I must have been in a daze 🙂

Categories: cloud Tags: , , , ,

Lewis Topographical Dictionary Ireland and SkyTwenty on EC2

November 17, 2010 Comments off

Both applications are now running on Amazon EC2 in a micro instance.

  • AMI 32bit Ubuntu 10.04 (ami-480df921)
  • OpenJDK6
  • Tomcat6
  • MySQL5
  • Joseki 3.4.2
  • Jena 3.6.2
  • Sesame2
  • Empire 0.7
  • DynDNS ddclient (see [1])

Dont try installing sun-java6-jdk, it wont work. You might get it installed if you try running instance as m1.small, and do it as the first task on the AMI instance. Didnt suit me, as I discovered too late, and my motivation to want to install it turned out to be no-propagation of JAVA_OPTS, not the jdk. See earlier post on setting up Ubuntu.

  • Lewist Topographical Dictionary of Ireland
    • Javascript/Ajax to sparql endpoint. Speedy.
    • Extraction and RDF generation from unstructured text with custom software.
    • Sparql endpoint on Joseki, with custom content negotiation
    • Ontology for location, roads, related locations, administrative description, natural resources, populations, peerage.
    • Onotology for Peerage – Nobility, Gentry, Commoner.
    • Find locations where peers have more than one seat
    • Did one peer know another, in what locations, degree of separation
    • Linked Open Data connections to dbPedia, GeoNames (uberblic and sindice to come) – find people in dbPedia born in 1842 for your selected location. Map on google maps with geoNames sourced wgs84 lat/long.
  • SkyTwenty
    • Location based service built JPA based Enterprise app on Semantic repo (sesame native).
    • Spring with SpringSec ACL, OpenID Authorisation.
    • Location and profile tagging with Umbel Subject Concepts.
    • FOAF and SIOC based ontology
    • Semantic query console – “find locations tagged like this”, “find locations posted by people like me”
    • Scheduled queries, with customisable action on success or failure
    • Location sharing and messaging with ACL groups – – identity hidden and location and date time cloaked to medium accuracy.
    • Commercial apps possible – identity hidden and location and date time cloaked to low accuracy
    • Data mining across all data for aggregate queries – very low accuracy, no app/group/person identifiable
    • To come
      • OpenAuth for application federation,
      • split/dual JPA – to rdbms for typical app behaviour, to semantic repo for query console
      • API documentation

A report on how these were developed and the things learned is planned, warts and all.

[1]http://blog.codesta.com/codesta_weblog/2008/02/amazon-ec2—wh.html – not everything needs to be done, but you’ll get the idea. Install ddclient and follow instructions.

Installing JDKs & Tomcat on Ubuntu 10.04/10 AMI on EC2

November 8, 2010 Comments off

That title is a mouthful isnt it ? A cloudful.  Only slightly better than “How to go about successfully installing Sun JDK6 on Ubuntu 10.04/10 (Lucid/Maverick) using an Ubuntu EBS image on Amazon EC2”

I am trying to setup an Ubuntu 10.04 image on EC2 as a t1.micro. Amazon are giving away free time for the first year on this size image so its a good way to get to grips with things and learn the ins and outs.

So

1) get an amazon account 🙂

2) sign up for EC2 – means handing over credit card details and read the quick getting started guide

2.1) download the cmdline tools or use the web interface.  There are some things you need this for because the web console doesnt have everything. I used the web interface to set up credentials and so on, had to set up ssh on cygwin, then switched to putty (use puttygen to convert the amazon private key first). Cmdline tools need a pk and a x509 cert, so you need to follow the instructions to create those as well.

3) find an image – I have unsuccessfully tried

  • ami-480df921 – a basic ubuntu i386 10.04 on EBS
  • ami-508c7839 – a basic ubuntu i286 10.10 on EBS

4) read some documentation usage docs about volumes, snapshots, reboots and termination and whatever you can from the library of docs and tutorials on  pricing, sizing, architecture

5) decide what you need installed for your purposes – you will install software on your instance just like its your own – that is to say there is no special amazon repo or methodology to follow

6) go for it – first change the instance so the root volume is preserved on reboot (otherwise you’ll lost it all)

/dev/sda1=vol-ddf910b5:attached:2010-11-15T17:09:44.000Z:true

6.1) I wanted Java, Tomcat, mySQL and my own software

6.1.1) Java & Tomcat – I used sudo termsel – and chose the Java-Tomcat option – what could be easier ? Well, lots actually. The task hangs, and no amount of ping, ssh or retries helps. So, reboot the instance : this seems to work.  Syslog has nothing to reveal. Try again, same thing happens. Turns out its trashed, so you need to use apt-get purge on the openjdk jre packages it attempts to install then do an apt-get update, then try again. It worked for me, but I suspect YMMV.

6.1.2) MySQL – use apt-get and install the MySQL-server package

6.1.3) my software – jarred up an instance of my webapp, WinSCPd to the instance (very slow), and unjarred. Had to install the OpenJDK (not jre) package to get jar of course. Sigh. Added Ubuntu to the tomcat6 group with adduser (not useradd !!!), copied and moved webapps and libs around the place, then fixed up local paths in my config.

Restart tomcat and cross fingers. My JNDI context name (jdbc) isnt being found and an instrumentation agent Im using with -agent on the JAVA_OPTS is failing. (getAllLoadedClasses() is returning null ! ) Could it be that I need the Sun JDK ?…..

6.1.4) Attempt to install Sun JDK. apt-get install sun-jdk6 you might guess – but thats too easy.  Sun JDK6 is now in an archive repository, so you need to add that

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
sudo apt-get update
sudo apt-get install sun-java6-jdk

and then, wait for it, your instance hangs. So, Im about to try to fix that, by following instructions from an Ubuntu launchpad bug report.

i) In theory, first thing to do is start your t.micro instance as m1.small. You need the cmdline tools for this:

ec2-run-instances -k your-pk-file-name -g your-security-group-name -t m1.small ami-image-id

for example

ec2-run-instances -k .ssh/uoc_kp -g uoc-test-1 -t m1.small ami-480df871

ii) connect to the instance and install java. you’ll need to use the same pk in whatever ssh client you are using. the install java with


sudo apt-get update
sudo apt-get install sun-java6-jdk

iii) stop instance

iv) start normally – that is without setting type to m1.small

However, when you do i) you end up with a new instance with a new volume. So how can you install the jdk on the volume you need ?? Ive asked, but got no answer….

So now I try an Amazon Linux image. This however doesnt have Sun JDK available to install, so I’ll try one more time with the openjdk – maybe this version will work

  • launch instance and connect
  • scp tarball of install from ubuntu instance (install public key from ubuntu user in .ssh/authorized_keys on Amazon Linux AMI)/
  • Link /var/lib/tomcat6 to home dir : ln -sf /var/lib/tomcat6 tomcat6
  • install tomcat6, tweak /etc/init.d/tomcat6.conf so that CATALINA_BASE points at /var/lib/tomcat6
  • add ec2-user to tomcat group
  • chmod 775 ~/tomcat6 so that tomcat group can write logs etc
  • mv mysql connector jar to $CATALINA_HOME/lib
  • install mysql : sudo yum install mysql-server.i386 – but how to set up root user ? more pain !
    • well – when you first run /etc/init.d/mysqld it tells you
      /usr/bin/mysqladmin -u root password ‘new-password’
      /usr/bin/mysqladmin -u root -h ip-10-100-10-100 password ‘new-password’
  • run tomcat6 …..
  • same problem – no instrumentation (my app wont run without)

S0, digging deeper, I find eventually that my JAVA_OPTS dont get passed with sudo. So I use -E, and they get as far as /etc/init.d/tomcat6, but no further. Looking at /usr/sbin/tomcat6 which is called by the init script, I see that /etc/tomcat6/tomcat6.conf is pulled in, and env vars can be created here, including JAVA_OPTS (which has the -javaagent JVM param).  And its writable by the tomcat group, so this is the best place to put it, as long as you wrap it in quotes.

So, now, tomcat6 appears to be starting, but I have other problems to solve, but at least theres progress. Snails pace, but progress.

[Update]

That would have been too easy tho. On Amazon Linux AMI I seemed to have all sorts of strange permissions problems. properties files couldnt be read, a directory couldnt be created. So I have given up, and have moved back to the ubuntu 10.04 install I have for now. Knowing that the JAVA_OPTS were a problem, and seeing that /etc/tomcat6/tomcat6.conf doesnt exist and is not used by /etc/init.d/tomcat6 on ubuntu, all you need to to get JAVA_OPTS is set them in your env and use sudo with -E.

So thats Tomcat6 running in upto 384m, and Joseki on 256m.  Wonder how they’ll fair…

For log4j, I needed to set a fully qualified path, or I get log create errors.

And then, finally, had to update the Security Group to let traffic thru on the right ports. Now just need to update the code to the latest level, start a fresh repo on Sesame, restart a few more times (sigh) and test the jsp/js code for machine specifics I may have left lying around. Almost there !