Archive

Posts Tagged ‘hardware’

Parsing m3u file for curl checks using AWK

July 28, 2016 Comments off

I recently had reason to make use of an m3u playlist file for an IPTV device. I found one on http://www.sattvhelp.com, a great resource for all kinds of sat and IPTV issues. IIRC I came across this after finding a post about a filmon.tv plugin at http://iptvlivestream.com/iptv/filmon-tv/.

In any case the m3u file contained lots of links to IPTV stations that were no longer available of not responding anymore so I wrote an awk script to parse the m3u and based on an expression execute and action that was a system() call. AWK is a great tool to use search expressions and logic on records in files but is a PITA to debug – and my other attempts at using getline didnt help either. M3U files are a sequence of paired records which makes grep and shell scripting innappropriate I think (which I did try firstly) and awk seems a better way, even if it needs a bunch of calls out to another process/shell.

See my post on sattvhelp.com for more (http://www.sattvhelp.com/forum/technomate-non-linux-chat/54223-iptv-channels-tm-f3-5-tm5402-m3-33.html#post149601) but heres the script if you need it to parse/validate any other kind of m3u playlist from time to time.

#!/usr/bin/awk -f
 BEGIN {
 FS="\n"
 print "#EXTM3U"
 }
 {
 if ( $0 ~ /^#EXTINF/ ) {
 ITM=$0
 #reset URL so it doesnt print the same one twice
 URL=""
 } else if ( $0 !~ /^#EXTINF/ ) {
 URL=$0
 #print "Found "URL
 }

#print ITM"\n"URL
 # curl --connect-timeout $SLEEP --output /dev/null --silent $u
 #
 #RC=system("curl --connect-timeout 5 --output /dev/null -silent" URL)

if ( URL != "" ) {
 cmd = "curl --head --location --fail --max-time 10 --connect-timeout 5 --head --output /dev/null -silent " URL
 #print "Calling : "cmd
 RC = system( cmd )
 #print "Return code : "RC
 if ( RC == 0 ) {
 print ITM"\n"URL
 }
 }

}
Advertisements

Arduino Mega ADK tty device

December 29, 2011 Comments off

Unlike what most of the documentation on the arduino site refers to, the Mega ADK board makes use of /dev/ttyACM0 and not /dev/ttyUSBn[1]. And in order to use it, you need the latest 1.0 version of the arduino dev kit, which has an updated RXTX library to support it, so don’t go updating it, like all the ubuntu install instructions eventually get around to telling you to do. You may also get into a rats nest of uninstalling tty and ftdi drivers.

So

  1. Get a JDK6 or 7 JDK (32 or 64 bit available)[2]
  2. Download the arduino dev kit (32/686 or 64 bit available)[3]
  3. Install all the avr stuff : 'sudo apt-get install avr-libc avrdude binutils-avr gcc-avr'
  4. Install the downloaded jdk then 'sudo update-alternatives --config java' to set the default JVM.[4]
  5.  Untar the arduino download to a dir of your choice (your home dir perhaps, as you dont want to be doing this as root)
  6.  Plug in the arduino board to a free USB port, and then run ./arduino from the install dir
  7.  Set the ADK board from Tools->Board menu
  8.  Set the serial port from Tools->Serial Port– it should already be done. If its not, or it is disabled, something is wrong with your installation. Check previous steps. Cry. Try again.
  9.  From File->Examples->Basic choose Blink.
  10.  Click the right arrow icon to upload to your board. Check pin 13 amber LED is blinking.

[1]http://www.arduino.cc/playground/Learning/Linux – see last para of “Included Dependencies”
[2]http://www.oracle.com/technetwork/java/javase/downloads/java-se-jdk-7-download-432154.html
[3]http://arduino.cc/en/Main/Software
[4]http://www.shinephp.com/install-jdk-7-on-ubuntu/

Bacula on Ubuntu 10.04

October 14, 2010 Comments off

Chose bacula to do backups even tho I have a RAID1 NAS.
1) Drives fail – one or both of mine have already had a few hiccups
2) tape lasts and lasts even if it is more expensive than other media these days
3) tar, dump or whatever are “standards”

Looked at quite a few backup programs, after discovering that my NAS (a QNAP 219p) doesnt support tape drives (the flavour of linux doesnt have scsi tape support or “mt”).

Want to use network mounts because rsync-ing across the network duplicates the storage requirements from the NAS to the backup device. Using a network mount I can do an occassional full backup then incremental daily – files arent changing that much in my SOHO setup.

1) It came with Tapeware/Yosemite which I had used a long time ago for distributed backup and quite liked. But running that on another machine and backing up across the network didnt work :
a) on a linux box it crashed the hardware – suspect some kind of IRQ conflict or otherwise with graphics card. Took out an old fax/modem – it helped but still bombed a few hours alter
b) on windows it just errors on every file on the network mounts (yosemite – or barracuda – dont recommend this kind of thing)

2) Bacula – opensource/freeware. Full featured, has a GUI “bat”. Install screwed up to begin with then couldn’t uninstall until I discovered that Ubuntu software manager doesnt uninstall all the components. Had to use apt to uninstall from the cmdline.
But it wouldnt work no matter what I tried and the config looked too daunting. Surely backup GUIs can be built for this kind of thing ? Gave up, for now – but see point X below
3) Amanda- big, GUI is commercial. Doesnt to tape append. Tape spanning wasteful apparently (cite). Looks daunting.
4) afbackup – 4 guis. cryptic config. Opensource documentation – ie very little
5) PCBackup – to disk not tape – excluded
6) tar and shell cmds – “i want pearls not cinders !” could do this but really want a database to manage catalog and restore.
7) Back to Bacula. Tinker some more. Look for documentation – better than most of the others, but still plenty of dead ends. BAT couldnt find my storage daemons. Config checked 4 times. Then changed from using hostname (specifically noted in config not to use ‘localhost’) to 127.0.0.1 and like magic things started to connect and be found. Must look at resolv.conf some time and see whats going on – think the hostname wasnt being resolved within timeout or hosts file being completely ignored for some reason. Could be something to do with domain config.

Worked out how to get my DAT72 tape device configured (it has a USB interface, not scsi, but linux takes care of the necessary spoofing) and then created a Pool, a Job and a FileSet. Wish there was a gui to pick directories and so on but its done now, and wasnt so hard.
BAT is a bit idiosyncratic, but it does work if you stick with it. Got a job running now to backup some samba mounts I setup to the NAS – hope it works.

And Incremental backups – dont setup an individual JobDef for level = incremental, use a full backup and change the level in a schedule you use it with. Otherwise, it will get “upgraded” to a full backup because the FileSet and Job name will match with nothing in the database (unless you run the incremental first and then again within a schedule – which could be another story….)

Nov 8 2010

Bacula (or my Tape drive) is not playing ball.

I had set up a job with scheduled weekly full with daily incremental, but also, had a scheduled power down and restart around it – lets not waste electricity just because we cant see it 🙂 But Ive found that doing a restart with “shutdown -h now” causes the tape drive (or perhaps one of the bacula init scripts) to write an MTEOM on the tape. When the next scheduled job comes along it then fails because of the MTEOM with “no available media” and asks for a new tape to be added to the pool and labelled.  The default ‘mt’ on Ubuntu doesnt seem to allow MTEOM control, I cant tell if the device driver is the cause, and havent had time to look into the bacula scripts. The bacula docs dont say much, and forum messages talk about faulty tapes or drives – one I found mentions the reboot being the cause, which it seems to be in this case. Erasing the tape fixes it, but thats no help when you want your data backed up and on tape in the first place !

Options

  • Investigate driver and mt
  • Investigate scripts
  • Try different tape drive
  • Try another backup program (again)
  • Change job

Update March 14 2011

Have just had a full backup oafter 2 warm boots and a cold boot, which has been impossible up to now. One of 2 things has done this :

1) an apt-get upgrade
2) update bacula-sd.conf definition for my tape device :

#
# A UOC tape drive
#
Device {
Name = DDS-5
Description = "non rewind DDS-5 for HP C7438A DAT72"
Media Type = DDS-5
Archive Device = /dev/st0
Autochanger = no
RemovableMedia = yes; #added March-14-2011
Random Access = no
AutomaticMount = yes;               # when device opened, read it
Requires Mount = no; # added March-14-2011
AlwaysOpen = yes
Offline On Unmount = no
Hardware End of Medium = no
#next 3  lines  added March-14-2011
Fast Forward Space File = No # This line required if above HEOM is set to "No"
BSF at EOM = no # Default is no
Two EOF = no
Spool Directory = /var/bacula/spool
Maximum Spool Size = 15G
}

This is thanks to a post [1] on BackupCentral.com about a Dell/IBM Tape Library (TS3100). Phew. If things change over the next while I’ll post some more here – this seems to be a pretty popular post on this blog.

 

Update March 28 2011

Had more errors, not dissimilar to before, for new jobs or “next day” jobs after a successful backup. Bacula always reports that tape isn’t readable because of ‘

ioctl MTWEOF error on "Drive-1"'

. Another post [2] makes me think this is just a hardware config issue to do with the USB-SCCI tape drive Im using, and because I’m taking a one-tape[3] backup strategy. So for now I’ve just changed to AlwaysOpen=no, so that the tape is released after every job. Maybe this will work, but I’ve checked (to no avail) for opensource linux tape backup software again, that has a gui and catalog, because Im getting bored with this. I still dont trust disk tho, even on a RAID box. Kit gets stolen, buildings burn down, and Im not prepared to go to the cloud for this – too slow and expensive for now.

[1] http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/bacula-25/tape-mteom-error-with-dell-tl2000-ibm-ts3100-97809/
[2] http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg25279.html
[3] http://www.bacula.org/fr/dev-manual/Backup_Strategies.html#SECTION001610000000000000000

EReader with GPS ?

June 21, 2010 Comments off

I want an eReader with

  • GPS
  • WIFI
  • browser
  • Flash support

Cheap, expandable, with a good developer API. I know the Dell Streak might come close, or an iPad, but I dont want to spend 300-800 smackeroos on one. B&W is good, big readable screen is good, standards support is good. Any suggestions ?