Archive

Posts Tagged ‘ubuntu’

Ubuntu upgrade to 11.04 (403 errors)

May 2, 2011 Comments off

If you decide to upgrade to 11.04 and are getting 403 Forbidden error messages when packages are being download (I got it for xulrunner) then try starting Update Manager, locate the settings for the mirror you are using and change it to something else – in my case I reset it to “main”. You wont have to download everything again as the files are cached, but you should get past the problem and your upgrade should continue and hopefully succeed (I had a lot of clicking to do as I was asked if I wanted to modify config files I had customised).

Categories: technology Tags: , , ,

Installing JDKs & Tomcat on Ubuntu 10.04/10 AMI on EC2

November 8, 2010 Comments off

That title is a mouthful isnt it ? A cloudful.  Only slightly better than “How to go about successfully installing Sun JDK6 on Ubuntu 10.04/10 (Lucid/Maverick) using an Ubuntu EBS image on Amazon EC2”

I am trying to setup an Ubuntu 10.04 image on EC2 as a t1.micro. Amazon are giving away free time for the first year on this size image so its a good way to get to grips with things and learn the ins and outs.

So

1) get an amazon account 🙂

2) sign up for EC2 – means handing over credit card details and read the quick getting started guide

2.1) download the cmdline tools or use the web interface.  There are some things you need this for because the web console doesnt have everything. I used the web interface to set up credentials and so on, had to set up ssh on cygwin, then switched to putty (use puttygen to convert the amazon private key first). Cmdline tools need a pk and a x509 cert, so you need to follow the instructions to create those as well.

3) find an image – I have unsuccessfully tried

  • ami-480df921 – a basic ubuntu i386 10.04 on EBS
  • ami-508c7839 – a basic ubuntu i286 10.10 on EBS

4) read some documentation usage docs about volumes, snapshots, reboots and termination and whatever you can from the library of docs and tutorials on  pricing, sizing, architecture

5) decide what you need installed for your purposes – you will install software on your instance just like its your own – that is to say there is no special amazon repo or methodology to follow

6) go for it – first change the instance so the root volume is preserved on reboot (otherwise you’ll lost it all)

/dev/sda1=vol-ddf910b5:attached:2010-11-15T17:09:44.000Z:true

6.1) I wanted Java, Tomcat, mySQL and my own software

6.1.1) Java & Tomcat – I used sudo termsel – and chose the Java-Tomcat option – what could be easier ? Well, lots actually. The task hangs, and no amount of ping, ssh or retries helps. So, reboot the instance : this seems to work.  Syslog has nothing to reveal. Try again, same thing happens. Turns out its trashed, so you need to use apt-get purge on the openjdk jre packages it attempts to install then do an apt-get update, then try again. It worked for me, but I suspect YMMV.

6.1.2) MySQL – use apt-get and install the MySQL-server package

6.1.3) my software – jarred up an instance of my webapp, WinSCPd to the instance (very slow), and unjarred. Had to install the OpenJDK (not jre) package to get jar of course. Sigh. Added Ubuntu to the tomcat6 group with adduser (not useradd !!!), copied and moved webapps and libs around the place, then fixed up local paths in my config.

Restart tomcat and cross fingers. My JNDI context name (jdbc) isnt being found and an instrumentation agent Im using with -agent on the JAVA_OPTS is failing. (getAllLoadedClasses() is returning null ! ) Could it be that I need the Sun JDK ?…..

6.1.4) Attempt to install Sun JDK. apt-get install sun-jdk6 you might guess – but thats too easy.  Sun JDK6 is now in an archive repository, so you need to add that

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
sudo apt-get update
sudo apt-get install sun-java6-jdk

and then, wait for it, your instance hangs. So, Im about to try to fix that, by following instructions from an Ubuntu launchpad bug report.

i) In theory, first thing to do is start your t.micro instance as m1.small. You need the cmdline tools for this:

ec2-run-instances -k your-pk-file-name -g your-security-group-name -t m1.small ami-image-id

for example

ec2-run-instances -k .ssh/uoc_kp -g uoc-test-1 -t m1.small ami-480df871

ii) connect to the instance and install java. you’ll need to use the same pk in whatever ssh client you are using. the install java with


sudo apt-get update
sudo apt-get install sun-java6-jdk

iii) stop instance

iv) start normally – that is without setting type to m1.small

However, when you do i) you end up with a new instance with a new volume. So how can you install the jdk on the volume you need ?? Ive asked, but got no answer….

So now I try an Amazon Linux image. This however doesnt have Sun JDK available to install, so I’ll try one more time with the openjdk – maybe this version will work

  • launch instance and connect
  • scp tarball of install from ubuntu instance (install public key from ubuntu user in .ssh/authorized_keys on Amazon Linux AMI)/
  • Link /var/lib/tomcat6 to home dir : ln -sf /var/lib/tomcat6 tomcat6
  • install tomcat6, tweak /etc/init.d/tomcat6.conf so that CATALINA_BASE points at /var/lib/tomcat6
  • add ec2-user to tomcat group
  • chmod 775 ~/tomcat6 so that tomcat group can write logs etc
  • mv mysql connector jar to $CATALINA_HOME/lib
  • install mysql : sudo yum install mysql-server.i386 – but how to set up root user ? more pain !
    • well – when you first run /etc/init.d/mysqld it tells you
      /usr/bin/mysqladmin -u root password ‘new-password’
      /usr/bin/mysqladmin -u root -h ip-10-100-10-100 password ‘new-password’
  • run tomcat6 …..
  • same problem – no instrumentation (my app wont run without)

S0, digging deeper, I find eventually that my JAVA_OPTS dont get passed with sudo. So I use -E, and they get as far as /etc/init.d/tomcat6, but no further. Looking at /usr/sbin/tomcat6 which is called by the init script, I see that /etc/tomcat6/tomcat6.conf is pulled in, and env vars can be created here, including JAVA_OPTS (which has the -javaagent JVM param).  And its writable by the tomcat group, so this is the best place to put it, as long as you wrap it in quotes.

So, now, tomcat6 appears to be starting, but I have other problems to solve, but at least theres progress. Snails pace, but progress.

[Update]

That would have been too easy tho. On Amazon Linux AMI I seemed to have all sorts of strange permissions problems. properties files couldnt be read, a directory couldnt be created. So I have given up, and have moved back to the ubuntu 10.04 install I have for now. Knowing that the JAVA_OPTS were a problem, and seeing that /etc/tomcat6/tomcat6.conf doesnt exist and is not used by /etc/init.d/tomcat6 on ubuntu, all you need to to get JAVA_OPTS is set them in your env and use sudo with -E.

So thats Tomcat6 running in upto 384m, and Joseki on 256m.  Wonder how they’ll fair…

For log4j, I needed to set a fully qualified path, or I get log create errors.

And then, finally, had to update the Security Group to let traffic thru on the right ports. Now just need to update the code to the latest level, start a fresh repo on Sesame, restart a few more times (sigh) and test the jsp/js code for machine specifics I may have left lying around. Almost there !

Bacula on Ubuntu 10.04

October 14, 2010 Comments off

Chose bacula to do backups even tho I have a RAID1 NAS.
1) Drives fail – one or both of mine have already had a few hiccups
2) tape lasts and lasts even if it is more expensive than other media these days
3) tar, dump or whatever are “standards”

Looked at quite a few backup programs, after discovering that my NAS (a QNAP 219p) doesnt support tape drives (the flavour of linux doesnt have scsi tape support or “mt”).

Want to use network mounts because rsync-ing across the network duplicates the storage requirements from the NAS to the backup device. Using a network mount I can do an occassional full backup then incremental daily – files arent changing that much in my SOHO setup.

1) It came with Tapeware/Yosemite which I had used a long time ago for distributed backup and quite liked. But running that on another machine and backing up across the network didnt work :
a) on a linux box it crashed the hardware – suspect some kind of IRQ conflict or otherwise with graphics card. Took out an old fax/modem – it helped but still bombed a few hours alter
b) on windows it just errors on every file on the network mounts (yosemite – or barracuda – dont recommend this kind of thing)

2) Bacula – opensource/freeware. Full featured, has a GUI “bat”. Install screwed up to begin with then couldn’t uninstall until I discovered that Ubuntu software manager doesnt uninstall all the components. Had to use apt to uninstall from the cmdline.
But it wouldnt work no matter what I tried and the config looked too daunting. Surely backup GUIs can be built for this kind of thing ? Gave up, for now – but see point X below
3) Amanda- big, GUI is commercial. Doesnt to tape append. Tape spanning wasteful apparently (cite). Looks daunting.
4) afbackup – 4 guis. cryptic config. Opensource documentation – ie very little
5) PCBackup – to disk not tape – excluded
6) tar and shell cmds – “i want pearls not cinders !” could do this but really want a database to manage catalog and restore.
7) Back to Bacula. Tinker some more. Look for documentation – better than most of the others, but still plenty of dead ends. BAT couldnt find my storage daemons. Config checked 4 times. Then changed from using hostname (specifically noted in config not to use ‘localhost’) to 127.0.0.1 and like magic things started to connect and be found. Must look at resolv.conf some time and see whats going on – think the hostname wasnt being resolved within timeout or hosts file being completely ignored for some reason. Could be something to do with domain config.

Worked out how to get my DAT72 tape device configured (it has a USB interface, not scsi, but linux takes care of the necessary spoofing) and then created a Pool, a Job and a FileSet. Wish there was a gui to pick directories and so on but its done now, and wasnt so hard.
BAT is a bit idiosyncratic, but it does work if you stick with it. Got a job running now to backup some samba mounts I setup to the NAS – hope it works.

And Incremental backups – dont setup an individual JobDef for level = incremental, use a full backup and change the level in a schedule you use it with. Otherwise, it will get “upgraded” to a full backup because the FileSet and Job name will match with nothing in the database (unless you run the incremental first and then again within a schedule – which could be another story….)

Nov 8 2010

Bacula (or my Tape drive) is not playing ball.

I had set up a job with scheduled weekly full with daily incremental, but also, had a scheduled power down and restart around it – lets not waste electricity just because we cant see it 🙂 But Ive found that doing a restart with “shutdown -h now” causes the tape drive (or perhaps one of the bacula init scripts) to write an MTEOM on the tape. When the next scheduled job comes along it then fails because of the MTEOM with “no available media” and asks for a new tape to be added to the pool and labelled.  The default ‘mt’ on Ubuntu doesnt seem to allow MTEOM control, I cant tell if the device driver is the cause, and havent had time to look into the bacula scripts. The bacula docs dont say much, and forum messages talk about faulty tapes or drives – one I found mentions the reboot being the cause, which it seems to be in this case. Erasing the tape fixes it, but thats no help when you want your data backed up and on tape in the first place !

Options

  • Investigate driver and mt
  • Investigate scripts
  • Try different tape drive
  • Try another backup program (again)
  • Change job

Update March 14 2011

Have just had a full backup oafter 2 warm boots and a cold boot, which has been impossible up to now. One of 2 things has done this :

1) an apt-get upgrade
2) update bacula-sd.conf definition for my tape device :

#
# A UOC tape drive
#
Device {
Name = DDS-5
Description = "non rewind DDS-5 for HP C7438A DAT72"
Media Type = DDS-5
Archive Device = /dev/st0
Autochanger = no
RemovableMedia = yes; #added March-14-2011
Random Access = no
AutomaticMount = yes;               # when device opened, read it
Requires Mount = no; # added March-14-2011
AlwaysOpen = yes
Offline On Unmount = no
Hardware End of Medium = no
#next 3  lines  added March-14-2011
Fast Forward Space File = No # This line required if above HEOM is set to "No"
BSF at EOM = no # Default is no
Two EOF = no
Spool Directory = /var/bacula/spool
Maximum Spool Size = 15G
}

This is thanks to a post [1] on BackupCentral.com about a Dell/IBM Tape Library (TS3100). Phew. If things change over the next while I’ll post some more here – this seems to be a pretty popular post on this blog.

 

Update March 28 2011

Had more errors, not dissimilar to before, for new jobs or “next day” jobs after a successful backup. Bacula always reports that tape isn’t readable because of ‘

ioctl MTWEOF error on "Drive-1"'

. Another post [2] makes me think this is just a hardware config issue to do with the USB-SCCI tape drive Im using, and because I’m taking a one-tape[3] backup strategy. So for now I’ve just changed to AlwaysOpen=no, so that the tape is released after every job. Maybe this will work, but I’ve checked (to no avail) for opensource linux tape backup software again, that has a gui and catalog, because Im getting bored with this. I still dont trust disk tho, even on a RAID box. Kit gets stolen, buildings burn down, and Im not prepared to go to the cloud for this – too slow and expensive for now.

[1] http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/bacula-25/tape-mteom-error-with-dell-tl2000-ibm-ts3100-97809/
[2] http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg25279.html
[3] http://www.bacula.org/fr/dev-manual/Backup_Strategies.html#SECTION001610000000000000000