State of Development - August 2018
We’re back with the State of Development for August. We have a lot of awesome stuff coming and can’t wait to share with you. Let’s hop in!
Table of Contents
- Controls (Gamepads, Mobile, Switch)
- Weapon Balance
- Building Effectiveness
- Player Support
- Account Security
- Playground Updates
Controls (Gamepads, Mobile, Switch)We believe you should be able to play Fortnite when and how you want, whether it's on the couch with a controller or on the go with Fortnite Mobile. We've identified and are working on various improvements and bug fixes related to aiming on controllers (including gyro controls). These include improvements to dead zones and accelerations to allow for better fine tuned aiming.
We have an upgrade to our matchmaking logic that will allow for a more even playing field when enjoying Fortnite with your input device of choice. In an upcoming release we’ll be implementing input based matchmaking, which will pair you against players who are using the same peripherals (e.g. controller vs controller, KB+M vs KB+M).
Let’s break that down:
- If anyone in your party uses a mouse and keyboard on console, you will be matchmade with PC players and other KB+M players (as if you partied up with a PC player).
The current crossplay rules still apply. For example, partying up with a PC player will always put you in a PC pool, regardless of your input type.
We are hoping to land this later in the year, and it will give you the ability to completely remap your gamepad to exactly your preference. In addition to customization, we’re continuing to hammer away at feel and optimizations for controllers across all platforms.
We dropped a balance blog detailing our logic behind recent changes. You can give that a read here. We’ve been through several phases in terms of weapon strength over the past few months, from the “double pump” strategy to the rise in players using SMGs. This is an area where we want to continuously evaluate and make adjustments to ensure different weapons have natural strengths and weaknesses in order to fulfill their intended role.
Some upcoming changes:
- Evaluating shotgun equip time. The equip time was originally added to help combat rapid weapon switching. However, since the shotgun switch delay was added, equip times have less of a purpose.
- We’ll be improving the Pump Shotgun so it’s a bit quicker to bring up, and we’ll be evaluating the other shotguns moving forward.
- Clingers are a bit more effective than we’d like.
- The behavior has been adjusted so that the second Clinger will fall to the ground in this scenario (v5.30).
We took our first steps into tweaking building with the material health adjustments in our v5.0 update. Last week we re-adjusted those knobs to more distinctly separate the types of building material, (metal stronger than brick, and brick stronger than wood) with the goal of creating more opportunities for tactical decision making.
We will continue to evaluate adjustments to building as the dust on all these changes settles. Following the moving storm changes with Season 5, we’re evaluating other potential ways to modify circle behavior, including how the storm might interact with player-built structures in the final circles.
Every update we’re working to improve your in-game experience. Lately we’ve been focusing on 50v50 mode, reducing long hitches and reducing streaming artifacts like low resolution buildings.
Keeping performance up to our standards is something that needs constant attention. Adding a new feature (for example, ATKs) comes with a performance cost, and we need to make sure everything we’re adding is efficient. Often we need to make savings in one area in order to cover the cost of new features. We monitor performance closely between releases (and before releases) to ensure we can detect and fix perf issues as soon as they’re found.
We’re working to improve your experience in 50v50 mode, where we tend to have a lot of players gathering in one area at one time. The graph below shows percentage of missed frames on Xbox One and PS4 (this is a comparison of actual number of frames rendered and the ideal number of frames rendered at 60fps) - lower is better. As you can see, performance in 50v50 steadily improved with every release except for the initial 5.0 release, where the addition of vehicles caused a temporary regression.
This is the same graph, but looking at missed frames at 30fps on other platforms.
It’s noticeable that both iOS and Switch have got worse in 50v50 mode since launch. This is something we’re currently looking into.
There’s still room for improvement to get 50v50 to a solid frame rate across all platforms, and this is an ongoing effort for the whole team. You can expect to see further improvements in the coming months.
Hitching (Screen Freezing)
We’ve heard players talking about hitches during their regular Fortnite play, and over the past few months we’ve been working to improve in that area. In particular, we’ve been focusing on long gameplay affecting hitches of half a second or more.
For the upcoming v5.30 update, we tracked down and fixed the cause of a very long (multi-second) hitch. This issue accounted for about 60% of our hitches over half a second in length. This was worst in long matches, especially Playground. It was particularly bad on Xbox One, but it also affected PS4, PC and Mac.
The graph below shows the average number of long (half second or longer) hitches per minute across the different releases on consoles.
This is the same graph, but for non-console platforms:
The significant reduction in Mac hitching in the v5.0 release was caused by improved caching for shader objects.
On iOS, runtime creation of shader objects remains the primary source of long hitches throughout the game and is something we are working to improve.
Slow Building LOD (Level of Detail) Streaming
We know how frustrating it can be when you land in Tilted and the buildings haven’t finished loading. A few releases ago, we added analytics so we can track this issue over time and detect regressions. Using this data (and your reports), we found that Xbox One and Switch are particularly bad in this area. We have some targeted improvements in the pipe for the v5.30 release which should improve the situation on these platforms.
Switch - Screen resolution
One of the complaints when we first released the Switch version of the game was that the screen was blurry in a lot of scenes. Switch shipped with the same deferred renderer that other consoles and PC use, and many of the settings were identical to Xbox One and PS4. We rely on a feature called dynamic resolution in order to prevent slow rendering from affecting the frame rate. This means that when the onscreen action gets too heavy, we’ll drop resolution instead of the framerate taking a hit. Obviously GPU efficiency is still important, because nobody wants to see blurry pixels all the time.
In v5.20 we moved to a new high-end forward renderer on Switch, which gave us an additional 3 milliseconds of GPU time (around a 10% improvement). This required a few engine changes, but crucially there was no visual fidelity loss. The end result is screen resolution is higher (and less blurry), and battery life should also be improved because the GPU isn’t having to work so hard.
For those of you playing Fortnite on a PS4 Pro, in the near future we're adding support for greater than 1080p res when connected to a 4K TV. Plan out that next drop with even more clarity!
Most of our regions have multiple datacenters supporting them. For example, our North America East region operates out of both Virginia and Ohio. Currently, matches are placed randomly on servers running in any datacenter within the region, which can lead to some variation in ping from one game to the next. To help with this, we’ve been working on sub-region matchmaking to allow us to route games to the datacenter in your region with the best ping for you. Once this change is live you should see much more consistent ping times across games.
Similarly, there are cases like our Asia region where players in Southeast Asia are not getting the kind of performance we’d like. Our solution is to add an additional datacenter in Singapore and use sub-region matchmaking to route you there if it would improve your ping, as long as wait times remain reasonable. When wait times grow too long, we’ll dynamically re-route you to a larger sub-region, in this case that’d be Tokyo. We think this is the best way to improve ping for the most people, without making ping worse for anyone.
Our goal is to provide the best possible quality of service everywhere in the world. We’ll continue to evaluate other areas where we can add additional data centers or new regions to improve performance for players. Some of the areas we’re investigating beyond Southeast Asia are the Middle East / Northern Africa and Russia. We’ll have more to share about our plans over the coming months.
Networked Player Movement / “Rubberbanding”
We’ve made headway on improvements to network corrections (i.e. rubberbanding) in v5.30, especially in the presence of unstable network connections.
There are a few common causes of bursts of packet loss, like WiFi interference (so try to use a wired connection), sub-optimal router firmware (upgrade to latest firmware if you can), ISP or datacenter level issues, or overloaded servers (becoming less of an issue as we optimize server performance). In cases where you have an unstable network connection (most notably bursts of packet loss*), this causes a few symptoms on your end as well as for other players that view your character.
*You can enable network debug stats via the settings menu in-game
On the local side for the client experiencing the packet loss, in-game, this can manifest as "rubberbanding" or warping of your location. This is due to divergence between the client and server simulations (based on input, velocity, etc). The server has final authority on where it thinks the client should be based on this simulation state, and if the error is unacceptably large, the server "corrects" the client to a new location. This actually happens on a small scale (a few centimeters) somewhat frequently because of normal network factors, for example collisions with other moving objects which may be slightly divergent (due to latency). Sometimes the corrections come from implementation bugs or intentional decisions to ease implementation. For example, entering a vehicle could just warp the player on the server and the client would get corrected to the correct location.
The Unreal Engine has built-in systems to smooth out these types of position corrections for players. This is fairly complicated and involves the client buffering saved moves that have not yet been acknowledged by the server (due to latency, since the client runs ahead of the server). When a correction is received the client replays local input after the corrected timestamp to try to get back to where you we moving (for example, if you were pressing forward for another 1/10th second after the corrected time, try to move forward there again). This covers up discrepancies between locally predicted movement and small server corrections quite well. For example, in very early versions of Fortnite some player actions such as targeting, melee swings, and jumping had timing-related logic bugs which occasionally caused a small correction for clients. These corrections were generally so small that the built-in systems could cover up the discrepancy and make it seamless to clients. However since there is some overhead to fixing up position and it could still sometimes be noticeable to players, we fixed all such known issues of small corrections in subsequent releases.
Now for some background on what happens when the server hasn't heard from a client for a long duration (e.g. greater than 0.25 seconds), usually due to bursts of packet loss: The server forces the server-side simulation of client movement forward in time based on the last known state (velocity etc), and this continues periodically until the client finally responds again. This is mainly intended to keep the client state moving for other observers (other remote clients), as well as to keep the server from getting too far behind in the simulation.
However during examination of what happens during long of packet loss burst, we discovered a few issues with the systems mentioned above. One issue was that these forced server updates were improperly advancing client timestamps, and this caused old outdated client packets to be accepted briefly on the server. The result was that the remote player experienced a series of server corrections inconsistent with the expected path along the current velocity. Additionally, after a long duration of network loss, the server forcing movement could cause a player to end up quite far from their expected location (either out from cover or over the side of a ledge), since simulation on the server continued without new input from the player during the connection disruption.
To address this we've improved the server code in the Unreal Engine that detects and handles this case to properly update client timestamps, as well as added a configurable max duration to these forced server adjustments. We've also included some helpful commands to simulate bursts of packet loss at various levels to help with testing these conditions. Look for these in upcoming UE 4.21 and later engine releases.
The result of our fixes is that after a burst of packet loss, the client location will be much more consistent with the expected change based on velocity state before the network interruption. In cases where the client was running forward during the entire disruption, the correction may not even be noticed. However it is possible that if the client was attempting drastic changes in direction or speed, there will be some warping, though it is still much improved over the old system. This also means that viewing other players having network issues will exhibit less warping as well since the trajectory will be more consistent. The duration limit also means that loss of control due to packet loss will move the character a limited distance, reducing the chance that they end up in the open or over a ledge after a sustained episode of network disruption, which could cause them to fall to their doom.
Proxy Simulation Improvements
Related to the above networked movement / rubberbanding improvements, we’ve also made adjustments to how we handle proxy simulation for both performance and quality. A “proxy” here means any player other than yourself that you see on-screen.
For performance, Fortnite has had various levels of movement simulation quality that it applies to proxies, and these have continued to be tweaked since initial release. The overall goal is basically to move many character proxies on your machine at minimal CPU cost. There is cost associated with updating transforms for all characters, their attached accessories (e.g. back bling), weapons, particle FX, and so on. One strategy to lower this cost is to simply move them less often or throttle the updates. The most obvious example is to avoid moving characters every frame when you can’t see because they’re out of your view angle. However you have to be careful here, because you might still need things like off-screen indicators or minimap indicators to have correct locations without jerky behavior. (Fortnite v5.20 included some fixes to bugs with the calculations to avoid throttled and jittery indicators, while still allowing us to throttle the expensive movement).
The downside of throttling movement updates is that at low update rates you can notice the jittery behavior of movement when motion doesn’t update every frame. Generally we have tuned our systems so that we try to throttle only characters that are far away or out of view. However special care has to be taken in some circumstances, such as zooming in through a scope at a character far away. Recent releases included fixes to lower or disable the throttling for characters in a small view angle at long range when using ADS or sniper scopes.
Also, since we initially created the system to throttle based purely on distance, this had the unfortunate side-effect of being overly aggressive in trying to save performance when only a few characters were relevant, which resulted in sacrificing some quality. Release v5.30 includes enhancements to also consider the number of relevant players, so that distance is not the only consideration. The end result is that when few players are relevant, even characters that are far away will have high-quality movement and interpolation. On the flip-side, in cases such as 50v50 where many characters are relevant, we try to reign in some quality settings in favor of improving client framerate. It’s a balancing act of trying to achieve better performance and improved quality, and work in this area as always is ongoing!
CompetitiveOur weekly Fortnite Summer Skirmish events have proven invaluable as a testing ground, providing a high-stakes environment for some of the best players in the world to showcase their skills. We have learned a lot from experiments with scoring schemes and formats, attempting to find the best balance between simplicity, entertainment, server performance and competitive integrity. Expect for us to continue operating and supporting competitive events after the conclusion of Summer Skirmish as we lead into next year's official Fortnite World Cup.
In recent weeks we began using a version of Live Spectating functionality similar to what was featured in the E3 Pro-Am but this time operating on live gameplay servers. Improvements have been made to allow for Gameplay POV cameras to more closely mirror the action when following a specific player. This will allow us to provide more high-quality coverage of the action along with map overheads, scoreboards, and drone cameras. We will be making extensive improvements to these features over the coming weeks and look forward to highlighting the performances of our competitors.
We are aware of a strong desire from competitive players for more opportunities to compete directly against one another within the same match. Development recently began for a robust competition system which will allow for all players to compete with one another and be recognized for their accomplishments. The first version of this feature is expected later this Fall.
MobileWe launched our Android Beta for Fortnite last week and we know it wasn’t the smoothest experience. We’re digging into the issues being reported by players on a variety of Android devices, and are working to resolve them. The main areas to address first will be stability and performance. Head over to our Trello board with the new mobile section tracking ongoing bug investigations.
One of our short term goals is to improve performance across the range of supported devices. Then beyond that, we want to support more devices in the beta and will hopefully open the beta to all min spec Android devices soon.
You can sign up for the Android Beta by visiting our website, check if you have a minimum spec device in our FAQ.
Additionally, getting you from your pocket to Paradise Palms as fast as possible is another issue we care about. We’re working on ways to make background downloading easier and dropping the overall patch size, so the only thing you have to focus on during patch day is where you’re dropping next.
Remember, you can only download Android via the official Epic Games website or through the Samsung website using game launcher.
Player SupportLet’s take a look at the current state of Player Support and where we need to be. We were not ready at launch for the number of players that Fortnite brought in and weren’t able to scale at a rate at which we consider up to our standards.
We’re working to grow and develop our Player Support Team as quickly as possible, but we’ve still got a way to go to achieve our standards of quality and service. Ramping up means creating learning materials for internal staff, implementing new tools and reacting to game events which cause an increase in player traffic. Our #1 goal is providing you with consistent, efficient, and high-quality player support.
In the past few months, we’ve dramatically grown the size of our Player Support Team. Planning ahead, it’s still not enough coverage for our incoming player needs. So, by the end of this year, we’ll grow that team size again.
# of Player Support agents
Ticket response times in some areas are bad, and we’re continually working to improve this by both increasing team size and upgrading the tools we use to communicate with you.
Behind the scenes, we’re rolling out and ramping up other tools that will allow our agents to respond to you faster. One tool we’ve updated is our CRM (customer relationship management) to one we believe will provide a better player experience. This new CRM will also allow our agents to work faster and more efficiently over time.
We’re continuing to hire experienced Player Support staff — seasoned directors, managers, agents, and partners. We’re still working to improve the team, but appreciate your continued patience and hold your satisfaction as our #1 goal.
Account security is one of our top priorities. We have a number of improvements in the pipeline that should help keep your account secure!
First, support for offline authenticators - (e.g. Google Authenticator, Microsoft Authenticator etc.) this will be available in the near future and will provide an alternative to our current email-based two factor authentication (2FA). We’re also considering support for additional two factor methods beyond authenticator apps, for example via SMS or in-app one click approval on mobile devices.
Second, improved detection around leaked credentials. Are you constantly getting emails about your locked account or with two factor authentication codes? If so, it’s likely that your password is being reused across multiple sites and was found in a password dump. Attackers use these password dumps to try and gain access to accounts beyond where the dump initiated - a so called “credential stuffing”. Epic has been implementing various layers of protection to mitigate these types of attacks. In case we do find a credential dump, we got your back. Our Information Security team constantly searches for new ones released on the internet and uses them to discover potentially vulnerable account credentials in an automated way. If a match is found, we notify you and issue a password reset to prevent unauthorized access. Collected dumps are anonymized and rehashed before use in our systems.
Lastly, we are actively reviewing and refining the state of security notifications you receive. The goal is to reduce the amount of 2FA or account lockout noise, while adding notification focus around unusual access patterns. We plan to improve protection around accessing or modifying account data, or disabling two factor authentication for that matter, by re-challenging with another 2FA check on each access attempt. We’re also actively working on improving account access recovery flows to help with cases when an account was actually hijacked. Account security is very important to us, and we’ll be working continually to improve internal tech and the resources we offer!
P.S. If you enable 2FA, you’ll unlock the Boogiedown Winner’s emote in-game for free!
All of the feedback for Playground has been amazing, we love seeing how folks can create and play on their own private island. We’re going to be working towards adding more specific knobs to allow you to dial in on your preferred Playground experience.
The next set of items dropping into Playground will be extra ATKs and more jumps around the island. Grab your squad, queue up, and see who can pull off the most impressive tricks. Keep sending us your thoughts on ideas for future versions!