Announcement

Collapse
No announcement yet.

Re-compressing Pak files (for 4.11) (discontinued)

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    [TUTORIAL] Re-compressing Pak files (for 4.11) (discontinued)

    Note:
    As of build 2883976 (2/29/2016), the pak files are compressed. There is no need to compress pak files manually anymore. However, this script is still applicable (and can also uncompress the pak files or make pak files compatible to newer builds).



    Since the pak files of the newer Unreal Engine are increased by its size, sharing files has been less enjoyable. A size of 300MB isn't that uncommon. The next build will fix that issue by compressing the pak files. If you can't wait for the next release, use the following script to re-compress an existing pak file. This can be done by simply dragging and dropping a file from the Windows explorer onto this script/batch file.

    Script
    • Download the CompressPak.bat file
    • Place that file into your Unreal Editor install location
      Code:
      C:\Program Files\Epic Games\UnrealTournamentEditor
    • Drag'n'drop a pak file from any location onto this file.
      (Note: This will by default overwrite that dropped file with the compressed pak file !!!)
    • Confirm by pressing any key to start the process



    Tech note
    The script file works as command line program as well.
    Code:
    CompressPak.bat <file> [Options]
    Options:
    -o force to overwrite the original file
    -f If option -o is not given, this will force to overwrite the new file if existing
    -u Uncompress the pak file instead of re-compressing it
    -s No output messages
    -n=<NUMBER>
    -n <NUMBER>
    Sets/overwrites the network version of the pak file
    Last edited by RattleSN4K3; 04-01-2016, 07:41 PM. Reason: New version 0.6
    ] Map Scaler Tool | Betrayal for UT4 | No Spawn Protection | No Pickup Timer | BioLauncher (revived) | ForcePickupSpawn | Map cosmetics::P | Safe Spawn::P | Why numbers for Health/Armor suck!::ANALYSIS/CONCEPT
    ] UT3 Addons: NoMoreDemoGuy | PickupRespawnTweak | Mutate Spec | MutePawnSounds | NoPlayerBeacon | Epic FTW | Epic FOCK | TripodSound (... and many more)

    #2
    :Wait
    if %script_no_wait% == 0 @ping 1.1.1.1 -n 1 -w %1 > nul
    goto :EOF

    why?

    you're also extracting pak files and not keeping the file order (this is important) so these paks are likely to break on some platforms.

    Comment


      #3
      Originally posted by raxxy View Post
      :Wait
      if %script_no_wait% == 0 @ping 1.1.1.1 -n 1 -w %1 > nul
      goto :EOF

      why?
      It was/is the most reliable waiting command. 127.0.0.1 or any equivalent doesn't work. If you know a better one without having to install/use external deps, let me know.

      Originally posted by raxxy View Post
      you're also extracting pak files and not keeping the file order (this is important) so these paks are likely to break on some platforms.
      Yep. Not sure if this is relevant as you say, therefore I left it out. I initially tried it but batch isn't that good for anything like parsing files/lines etc. By looking at the UnrealPak source, "-test" would result into a pre-sorted (respectively unsorted) output. If you know a proper way to iterate the output, I could change that.
      Code:
      LogPakFile:Display: Checking pak file "C:\Program Files\Epic Games\UnrealTournamentDev\UnrealTournament\Content\Paks\CTF-Grudge-R3M-v5-WindowsNoEditor.pak". This may take a while...
      LogPakFile:Display: "UnrealTournament/Content/RestrictedAssets/Blueprints/Lift/grudge1.uasset" OK.
      LogPakFile:Display: "UnrealTournament/CTF-Grudge-R3M-v5-AssetRegistry.bin" OK.
      LogPakFile:Display: "UnrealTournament/CTF-Grudge-R3M-v5-version.txt" OK.
      LogPakFile:Display: "UnrealTournament/Content/GTFreeMaterials/Textures/Ground_guiFloodedMud/Ground_guiFloodedMud_2k_alb.uasset" OK.
      LogPakFile:Display: "UnrealTournament/Content/RestrictedAssets/Environments/Tuba/Materials/M_TubaWater.uasset" OK.
      LogPakFile:Display: "UnrealTournament/Content/Command/HighresScreenshot00058.uasset" OK.
      LogPakFile:Display: Pak file "C:\Program Files\Epic Games\UnrealTournamentDev\UnrealTournament\Content\Paks\CTF-Grudge-R3M-v5-WindowsNoEditor.pak" healthy, 52 files checked.

      Originally posted by raxxy View Post
      [PAK ORDER] (this is important) so these paks are likely to break on some platforms.
      Are you sure about that?


      Edit:
      Regarding wait:
      Code:
      ping -n %1 127.0.0.1 > NUL
      with passing seconds seems to work. Pretty sure it wasn't initially I used that snippet on different OS/computers years back.
      Last edited by RattleSN4K3; 02-11-2016, 10:51 PM.
      ] Map Scaler Tool | Betrayal for UT4 | No Spawn Protection | No Pickup Timer | BioLauncher (revived) | ForcePickupSpawn | Map cosmetics::P | Safe Spawn::P | Why numbers for Health/Armor suck!::ANALYSIS/CONCEPT
      ] UT3 Addons: NoMoreDemoGuy | PickupRespawnTweak | Mutate Spec | MutePawnSounds | NoPlayerBeacon | Epic FTW | Epic FOCK | TripodSound (... and many more)

      Comment


        #4
        It repacks the map file all right but the map doesn't show up in the map list anymore.. loading the map with "open mapname" just goes to a grey screen.

        Edit: Repacked 7 maps with the script... One (Malevolence) works, the other 6 do not. All of them worked fine before repacking.
        Last edited by MoxNix; 02-11-2016, 10:55 PM.

        Comment


          #5
          Interesting if it works as advertised - Can I ask, what precisely does it do?
          I'm guessing it extracts the files out of the pak, then repaks the files using UnrealPak with compression enabled?
          ON-GOING PROJECTS: DM-BloodCovenant, DM-Campgrounds (Absolute)
          CONTACT & TWITCH: Absolute Discord
          , Twitch

          Comment


            #6
            Originally posted by MoxNix View Post
            It repacks the map file all right but the map doesn't show up in the map list anymore.. loading the map with "open mapname" just goes to a grey screen.

            Edit: Repacked 7 maps with the script... One (Malevolence) works, the other 6 do not. All of them worked fine before repacking.
            I also found a map where the extracted files do have a different structure and therefore the files need to be re-mapped properly. I currently have a version and testing it (with CTF-Grudge).

            Originally posted by Trinatek View Post
            Interesting if it works as advertised - Can I ask, what precisely does it do?
            I'm guessing it extracts the files out of the pak, then repaks the files using UnrealPak with compression enabled?
            Yes. It uses UnrealPak to extract and re-pack the files with that flag set.

            A proper way (they way I am current cooking), would be using the editor automation tool.
            Code:
            Engine\Build\BatchFiles\RunUAT.bat makeUTDLC -DLCName=MAP_NAME_WITHOUT_EXTENSION -platform=Win64 -version=3008042 -compressed
            Last edited by RattleSN4K3; 02-12-2016, 12:58 PM.
            ] Map Scaler Tool | Betrayal for UT4 | No Spawn Protection | No Pickup Timer | BioLauncher (revived) | ForcePickupSpawn | Map cosmetics::P | Safe Spawn::P | Why numbers for Health/Armor suck!::ANALYSIS/CONCEPT
            ] UT3 Addons: NoMoreDemoGuy | PickupRespawnTweak | Mutate Spec | MutePawnSounds | NoPlayerBeacon | Epic FTW | Epic FOCK | TripodSound (... and many more)

            Comment


              #7
              Originally posted by RattleSN4K3 View Post
              It was/is the most reliable waiting command. 127.0.0.1 or any equivalent doesn't work. If you know a better one without having to install/use external deps, let me know.



              Are you sure about that?

              Unless it was fixed by epic (they do file lists/orders for a reason on paks) then yeah, pretty sure.

              Back when I was running servers, giving everyone somewhere to host all their ****, i had to go to EXTENSIVE efforts to repack stuff and strip the asset bin (from 15MB to 1KB) and keep file orders the same.


              I do have a better wait method but not with access to it from work. will grab tonight

              Comment


                #8
                Updated the script. v0.3 is the most recent now. It should support that special mounting case which was existing in CTF-Grudge for instance.



                Originally posted by MoxNix View Post
                Edit: Repacked 7 maps with the script... One (Malevolence) works, the other 6 do not. All of them worked fine before repacking.
                Can you name the ones who are not working? And/or try these with the new version.
                ] Map Scaler Tool | Betrayal for UT4 | No Spawn Protection | No Pickup Timer | BioLauncher (revived) | ForcePickupSpawn | Map cosmetics::P | Safe Spawn::P | Why numbers for Health/Armor suck!::ANALYSIS/CONCEPT
                ] UT3 Addons: NoMoreDemoGuy | PickupRespawnTweak | Mutate Spec | MutePawnSounds | NoPlayerBeacon | Epic FTW | Epic FOCK | TripodSound (... and many more)

                Comment


                  #9
                  I could name them but none of them have been released yet, except DM-Malevolence-V11 and that's the one that works after repacking.

                  New version? Did you replace the file in the original link? If so I can try it.

                  Edit: I see you did update the file... I'll try it in a little while... As soon as the uncompressed repack (a long one) I'm doing finishes.
                  Last edited by MoxNix; 02-12-2016, 12:23 AM.

                  Comment


                    #10
                    Originally posted by MoxNix View Post
                    I could name them but none of them have been released yet, except DM-Malevolence-V11 and that's the one that works after repacking.
                    You can use the command line startup for the automation tool posted in #7. It is basically the same as what the upcoming fix does. This command line is the same as the editor one when you click on "Share".

                    Originally posted by MoxNix View Post
                    New version? Did you replace the file in the original link? If so I can try it.
                    Yes. The link is always the most recent one. Same file, same link, new versions. It is a Raw link to the one in the GitHub repo.
                    ] Map Scaler Tool | Betrayal for UT4 | No Spawn Protection | No Pickup Timer | BioLauncher (revived) | ForcePickupSpawn | Map cosmetics::P | Safe Spawn::P | Why numbers for Health/Armor suck!::ANALYSIS/CONCEPT
                    ] UT3 Addons: NoMoreDemoGuy | PickupRespawnTweak | Mutate Spec | MutePawnSounds | NoPlayerBeacon | Epic FTW | Epic FOCK | TripodSound (... and many more)

                    Comment


                      #11
                      Originally posted by RattleSN4K3 View Post
                      You can use the command line startup for the automation tool posted in #7. It is basically the same as what the upcoming fix does. This command line is the same as the editor one when you click on "Share".
                      Actually the holdup is I need to wait for the uncompressed pak to finish so I have a file to try repacking again.

                      Comment


                        #12
                        That v0.3 seems to have done the trick. The first 2 maps I tried repacking with the new version seem ok now. They show in the map list again and seem to play fine too.

                        I'll have to rebuild the other maps before I can try your script on them, but everything is good so far.

                        Comment


                          #13
                          Released a new version (0.5, link in the first post) to attempt preserving the pak order. There are some fixes, changes and additions to the script. So far, it is working. I am not sure if pushing these packages to clients work, and as raxxy said, if these works on different platforms (Linux, Mac). Use this tool on your own risk and only if you really want to share your map with others and reducing the file size.


                          Tech note regarding pak order:
                          By checking the CookerOpenOrder.log and the resulting pak file, UnrealPak doesn't provide a reliable pak order (test and list). Not sure if there is a proper way of doing a pak order with the given tools (on everyone's Windows system).


                          @MoxNix
                          Great to know.
                          Last edited by RattleSN4K3; 02-12-2016, 03:36 PM.
                          ] Map Scaler Tool | Betrayal for UT4 | No Spawn Protection | No Pickup Timer | BioLauncher (revived) | ForcePickupSpawn | Map cosmetics::P | Safe Spawn::P | Why numbers for Health/Armor suck!::ANALYSIS/CONCEPT
                          ] UT3 Addons: NoMoreDemoGuy | PickupRespawnTweak | Mutate Spec | MutePawnSounds | NoPlayerBeacon | Epic FTW | Epic FOCK | TripodSound (... and many more)

                          Comment


                            #14
                            Octagon (repacked with v0.3 of this script) is up on google drive now. Works fine here, but if anyone wants to test on different platforms and such, here's the link

                            https://drive.google.com/uc?export=d...E13UnBQdlM3Ums

                            The pak went from nearly 500 MB to just over 200 MB after repacking with the script.

                            Comment


                              #15
                              RattleSN4K3 There are things I'd like to do to you right now if you know what I'm sayin But ... ahem... I guess saying thanks will have to suffice for now. THANKS!!!!
                              Contact me: (Steam: Neillithan) (E-mail: neilvmoore@gmail.com)

                              Comment

                              Working...
                              X