Javascript is not a good solution here, because your page will fall apart for anyone who turns it off.
I think you can use
SSI (server side includes), although I've never used them, so I don't really know how well they work. They are probably your best bet with pure
HTML pages though.
The other thing you can do (bit of a long route) is rename all your files with a .php extension, and use the php include() function to include them. For example, you save the 6 divs you mentioned above, and you save them in 'divs.php'. Then in your main document, at the place they should exist, you use this function:
include("divs.php")
This will bring those divs into your code. But you have a document with a .php extension instead of a .html(l) extension. So you use a mod rewrite in your .htaccess file so that whenever someone looks for ___.htm, it says that address in the address bar, but the page they see is ___.php. They don't know its php though. If you need help with that, I think I have the necessary code in a file here somewhere, I can help you with it.
The third option I can think of is to use your .htaccess file, and add some code into it so that it processes .html documents as php. So they look like an .html document to the user, but have the functionality of php, which means you can use the php include() function in it. I have this .htaccess code as well if you want it.
I think the first option is the best, though I can't offer much assistance on it.
The second option is probably the the next best, but the most difficult to implement.
The third option is probably the easiest (after SSI), but you will take a hit in the speed of your .html pages, because instead of being served to the browser as is, they need to be run though the php parser.