Showing posts with label computer related stuff. Show all posts
Showing posts with label computer related stuff. Show all posts

01 February 2012

DHCP IP end is not in the subnet or not autorized

Yesterday night the Sagem bridge supplied by my ISP suddenly decided to reboot around 4:00.
Remote firmware update? I don't think so, the firmware build number seems pretty old.
Anyways, afterwards dhcp wasn't working anymore.
Turns out the dhcp server configuration was changed to serve 192.168.1.100 to 192.168.1.63...
Trying to change the .63 to something like 120 simply didn't work, the damn thing refuses to accept anything over 63... "DHCP IP end is not in the subnet or not autorized".
So I changed the dhcp config to serve 192.168.1.10-50, now it works again.

Firmware-VTU-R:5.5.1.2IK105012 Time Oct 1 2009, 14:04:47

31 January 2012

Why du and df report different disk usage.

Two different programs with two different goals.
du: diskusage, df: diskfree.
du is used to see how much space is used by files and folders on a file system.
df is used to see how much space is used/free on a partition.
Big difference.

du and df will report different disk usage and there are two main reasons.
Indeed, there are two main reasons and for some arcane reason when I hear people talking about this subject they almost never mention both possibilities.

du calculates the amount of space used by stuff on the file system by (special) files and directories.
df looks at the file system usage: how many blocks are available and how many blocks are used.
As file systems use some disk space to store meta data about the stuff on the file system, so even on an 'empty' file system df will still report a usage of a couple of MB, depending on the options chosen at creation time of course.
That metadata consists of inode tables, superblocks, extended attributes etc.
Those bits have to be stored somewhere, hence the difference in reported size.

The other reason they can differ in output is open file descriptors.
Let's say you edit a file and delete it from another terminal.
df won't report the used size as the inode isn't pointing anymore to the file (it was dereferenced after all), but du will still include the file in its calculation as the file is still open.
This situation is also valid in case where defunct processes are still keeping files open which are since long deleted.

Disclaimer: this is at least true on GNU/Linux and *BSD systems.

30 January 2012

Linux: about swap


I can't even begin to count how many time I've heard this argument: "You don't really need swap space as in an ideal situation your system shouldn't be swapping anyhow."

Well...*sigh*... no.

Linux has this little cool thing called swapiness; it's a number between 0 and 100, basically telling Linux how fast it should swap out unused applications.
That's right: how fast... not if.
Setting swapiness to 0 will result in Linux avoiding swapping out stuff as long as possible, turn it up to 100 and Linux will swap out agressively, trying to have as much free physical memory available at all time.
Everything in between 0 and 100... well, you get the picture.

Swapping out stuff is a good thing!
I'm not going to start a crusade to set swapiness to 100, but c'mon... don't tell me an application running 24/7 but only occasionally actually doing something should keep all its data in physical memory.
I like the idea of Linux swapping it out to disk, freeing valuable and expensive physical memory for something important in need of fast memory space.
Have a look on your system right now, do a  cat of /proc/sys/vm/swappiness.
I'm willing to bet it's not set to 0.

What I'm trying to say here is swap is a cool system, but it depends on what you're building.
Some systems can benefit greatly of having swap space at their disposal, sometimes you just want to run everything in RAM all the time.

Oh and btw: not having swap space at all is fucking retarded (we're talking desktops and servers here, not embedded stuff).

03 January 2012

PS3 Mediatomb Debian configuration

After writing the previous article about the wol file server set up I realized Mediatomb on Debian doesn't, by default, play nice with the PS3.
Here's my config file, it has a couple of extra mappings to make sure everything can be played on the PS3.
Don't forget to adjust the <home> bit and, if required, enable the YouTube and transcoding settings.


<?xml version="1.0" encoding="UTF-8"?>
<config version="2" xmlns="http://mediatomb.cc/config/2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://mediatomb.cc/config/2 http://mediatomb.cc/config/2.xsd">
  <server>
    <ui enabled="yes" show-tooltips="yes">
      <accounts enabled="no" session-timeout="30">
        <account user="mediatomb" password="mediatomb"/>
      </accounts>
    </ui>
    <name>Media</name>
    <udn>uuid:dae13796-e78c-486c-ab54-f79c144a20f3</udn>
    <home>/path/to/</home>
    <webroot>/usr/share/mediatomb/web</webroot>
    <storage caching="yes">
      <sqlite3 enabled="no">
        <database-file>sqlite3.db</database-file>
      </sqlite3>
      <mysql enabled="yes">
        <host>localhost</host>
        <username>mediatomb</username>
        <database>mediatomb</database>
      </mysql>
    </storage>
    <protocolInfo extend="yes"/>
    <custom-http-headers>
      <add header="X-User-Agent: redsonic"/>
    </custom-http-headers>
    <manufacturerURL>redsonic.com</manufacturerURL>
    <modelNumber>105</modelNumber>
    <extended-runtime-options>
      <ffmpegthumbnailer enabled="yes">
        <thumbnail-size>128</thumbnail-size>
        <seek-percentage>5</seek-percentage>
        <filmstrip-overlay>yes</filmstrip-overlay>
        <workaround-bugs>no</workaround-bugs>
        <image-quality>8</image-quality>
      </ffmpegthumbnailer>
      <mark-played-items enabled="no" suppress-cds-updates="yes">
        <string mode="prepend">*</string>
        <mark>
          <content>video</content>
        </mark>
      </mark-played-items>
    </extended-runtime-options>
  </server>
  <import hidden-files="no">
    <scripting script-charset="UTF-8">
      <common-script>/usr/share/mediatomb/js/common.js</common-script>
      <playlist-script>/usr/share/mediatomb/js/playlists.js</playlist-script>
      <virtual-layout type="builtin">
        <import-script>/usr/share/mediatomb/js/import.js</import-script>
      </virtual-layout>
    </scripting>
    <mappings>
      <extension-mimetype ignore-unknown="no">
        <map from="mp3" to="audio/mpeg"/>
        <map from="ogg" to="application/ogg"/>
        <map from="asf" to="video/x-ms-asf"/>
        <map from="asx" to="video/x-ms-asf"/>
        <map from="wma" to="audio/x-ms-wma"/>
        <map from="wax" to="audio/x-ms-wax"/>
        <map from="wmv" to="video/x-ms-wmv"/>
        <map from="wvx" to="video/x-ms-wvx"/>
        <map from="wm" to="video/x-ms-wm"/>
        <map from="wmx" to="video/x-ms-wmx"/>
        <map from="m3u" to="audio/x-mpegurl"/>
        <map from="pls" to="audio/x-scpls"/>
        <map from="flv" to="video/x-flv"/>
        <map from="avi" to="video/divx"/>
        <map from="mkv" to="video/x-matroska"/>
        <map from="mts" to="video/mpeg"/>
        <map from="ts" to="video/mpeg"/>
        <map from="m2ts" to="video/mpeg"/>
        <map from="mov" to="video/x-quicktime"/>
        <map from="vob" to="video/mpeg"/>
        <map from="m4v" to="video/mp4"/>
      </extension-mimetype>
      <mimetype-upnpclass>
        <map from="audio/*" to="object.item.audioItem.musicTrack"/>
        <map from="video/*" to="object.item.videoItem"/>
        <map from="image/*" to="object.item.imageItem"/>
      </mimetype-upnpclass>
      <mimetype-contenttype>
        <treat mimetype="audio/mpeg" as="mp3"/>
        <treat mimetype="application/ogg" as="ogg"/>
        <treat mimetype="audio/x-flac" as="flac"/>
        <treat mimetype="image/jpeg" as="jpg"/>
        <treat mimetype="audio/x-mpegurl" as="playlist"/>
        <treat mimetype="audio/x-scpls" as="playlist"/>
        <treat mimetype="audio/x-wav" as="pcm"/>
        <treat mimetype="video/x-msvideo" as="avi"/>
        <treat mimetype="video/quicktime" as="mov"/>
        <treat mimetype="video/x-quicktime" as="mov"/>
      </mimetype-contenttype>
    </mappings>
    <online-content>
      <YouTube enabled="no" refresh="28800" update-at-start="no" purge-after="604800" racy-content="exclude" format="mp4" hd="no">
        <favorites user="mediatomb"/>
        <standardfeed feed="most_viewed" time-range="today"/>
        <playlists user="mediatomb"/>
        <uploads user="mediatomb"/>
        <standardfeed feed="recently_featured" time-range="today"/>
      </YouTube>
    </online-content>
  </import>
  <transcoding enabled="no">
    <mimetype-profile-mappings>
      <transcode mimetype="video/x-flv" using="vlcmpeg"/>
      <transcode mimetype="application/ogg" using="vlcmpeg"/>
      <transcode mimetype="application/ogg" using="oggflac2raw"/>
      <transcode mimetype="audio/x-flac" using="oggflac2raw"/>
    </mimetype-profile-mappings>
    <profiles>
      <profile name="oggflac2raw" enabled="no" type="external">
        <mimetype>audio/L16</mimetype>
        <accept-url>no</accept-url>
        <first-resource>yes</first-resource>
        <accept-ogg-theora>no</accept-ogg-theora>
        <agent command="ogg123" arguments="-d raw -f %out %in"/>
        <buffer size="1048576" chunk-size="131072" fill-size="262144"/>
      </profile>
      <profile name="vlcmpeg" enabled="no" type="external">
        <mimetype>video/mpeg</mimetype>
        <accept-url>yes</accept-url>
        <first-resource>yes</first-resource>
        <accept-ogg-theora>yes</accept-ogg-theora>
        <agent command="vlc" arguments="-I dummy %in --sout #transcode{venc=ffmpeg,vcodec=mp2v,vb=4096,fps=25,aenc=ffmpeg,acodec=mpga,ab=192,samplerate=44100,channels=2}:standard{access=file,mux=ps,dst=%out} vlc:quit"/>
        <buffer size="14400000" chunk-size="512000" fill-size="120000"/>
      </profile>
    </profiles>
  </transcoding>
</config>

02 January 2012

File server WoL

A while ago I decided to upgrade my file server hardware but didn't want it to run 24/7 anymore.
Obviously I only use it for a small portion of the day, so it shouldn't be running all the time.
Saves money and energy, right?
I quickly decided to look into wake-on-lan to wake it up using an app on my phone.
The minor inconvenience of having to push a button on my phone and waiting a couple seconds would be offset by the saved money.

There are plenty WoL apps on the Android market, but I settled on "Wol Wake on Lan Wan", the main reason being the simple straightforward widget it has.
Push the widget and it sends the magic packet, that's it... works like a dream.

Server side, there were a couple of problems.
I use encryption for the storage volumes so just booting the server each time wasn't an option.
Having to enter the key each time kind of retarded for file server usage.
So that left me with a couple options:

  • Use a USB device with the decryption key on.
    • Insecure, physical access means access to the encrypted data.
  • Don't use encryption
    • Insecure, obviously.
  • suspend-to-disk / hibernate / S4.
    • Insecure, physical access means access to the encrypted data as hiberation writes the complete content of the memory to disk and then completely shuts down the server. Next time it boots it just puts the content back to memory.
    • Relatively slow.
  • suspend-to-ram / sleep / S3.
    • More secure. S3 doesn't write anything to disk, the system goes into a power saving state where everything but a couple components still get power to ensure the system can boot to full operational state quickly.
    • Less energy saving than the other options.
    • Wakes up very quickly (matter of seconds).

In the end, I went for S3 as it'd be secure enough for my needs, the power saving would still be a lot better than running it 24/7 and usability is exactly what I wanted.

For some reason the suspend wouldn't work correctly with a 2.6 kernel, so I installed 3.0.0 through apt-get.
It would go into sleep but never wake or work fine for a couple of times before stopping to work.
After troubleshooting the hardware I decided it must have had something to do with the kernel/LKM's so I tried the latest and greatest and it worked fine.
ACPI Linux implementation redesign? Beats me and tbh I've been too lazy to check out what exactly went wrong. Linux 3.0 works, so I'll be using it.

Moving on; the main focus of this project was to have an easy way to serve the files to my PS3 through Mediatomb.
I first thought about a system where a packet sniffer would constantly be sniffing the network for possible connections to the file server and send wake it up as soon as it saw any.
But that required a constantly running server and I wanted as less devices to run 24/7 as possible.
So then I though it'd be pretty cool to have the file server itself look if anything was connected to it and go to S3 state once a certain timeout when nothing connecting to it would be reached.
This is the script I'm using right now to do just that:
--------------------------------------------------
#!/bin/bash
timer=0
timeout=4
interval=60
while [ True ]; do
sync
date=`date`
echo "$date - MARK" >> /var/log/night
# If we're coming out of S3, we want to kill these
killall s2ram 2>/dev/null
killall pm-suspend 2>/dev/null
# Sometimes resyncing gets stuck on Debian
grep resync=PENDING /proc/mdstat 2>/dev/null
if [ $? -eq 0 ]; then
date=`date`
echo "$date - Setting md0 r/w" >> /var/log/night
mdadm --readwrite /dev/md0
fi
# Check our networking status
ping -c 1 192.168.1.1 2>/dev/null
if [ $? -ne 0 ]; then
date=`date`
echo "$date - Couldn't reach gateway, restarting networking" >> /var/log/night
/etc/init.d/networking restart
fi
# Check the WoL setting
ethtool eth2 | grep "Wake-on: g"
if [ $? -ne 0 ]; then
date=`date`
echo "$date - WoL setting wrong, correcting" >> /var/log/night
/sbin/ethtool -s eth2 wol g
fi
connections=`netstat -natp | grep ESTABLISHED | wc -l`
udpconnections="0"
if [ $connections == "0" -a $udpconnections == "0" ]; then
# Ok, no connection, let's increase the timer value
date=`date`
echo "$date - No connections, pass $timer of $timeout" >> /var/log/night
let timer=$timer+1
sleep 10
else
# Oops, there's a connection, resetting the timer
date=`date`
echo "$date - Connections detected" >> /var/log/night
timer=0
fi
# It's time to go to sleep
if [ $timer == $timeout ]; then
date=`date`
echo "$date - Going night night" >> /var/log/night
timer=0
pm-suspend &
sleep 30
fi
date=`date`
echo "$date - Lingering for $interval seconds" >> /var/log/night
sleep $interval
done
---------------------------------------

Of course, this script applies to my situation.
If you want to use it, you'll have to adjust some settings like the networking interface, the timeout you want, the ip address of the server itself and so on.
But it works pretty well and manages to avoid some pitfalls I encountered along the way.

Right, so the script was ready and worked as it should.
Now how could I get it to run "constantly"?
I tried running it through a startup script as a daemon, but for some reason it wouldn't survive the S3.
I tried running it through cron, which was fine but it wouldn't run 'all the time' as I originally wanted.
Even if the above solution would've been fine, it still wouldn't be optimal as the daemon/script could've been killed by other processes and I don't want that. I want it running, all the time.
So the perfect solution was to use init(8) through /etc/inittab:

hy:2345:respawn:/usr/local/sbin/hydragoesnightynight

The respawn line makes it respawn whenever its not running, no  matter what.
Even if the OOM killer should kill it, it'll be restarted by init and if init(8) gets killed off, there's a far bigger problem going on so it's perfect for my set up.
Anyways, initialize the new line with init q and it works.

The setup has been running for a couple months now and it suits my needs, YMMV of course.

Edit: refer to Label: wurmd for an easy way of waking the box back up.

For completeness' sake:
I'm running a combination of Debian testing and sid.
Linux: 3.0.0-1-486 (486 as I used the same disks as in my previous x86 compatible hardware).
NIC:  Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12).