New issue
Advanced search Search tips

Issue 638588 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 2
Type: Bug



Sign in to add a comment

OSX Gamepad API reports no buttons on SteelSeries Nimbus Controller

Project Member Reported by a...@figma.com, Aug 17 2016

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2830.0 Safari/537.36

Steps to reproduce the problem:
1. Use a SteelSeries Nimbus Controller
2. Use the gamepad API
3. Check the gamepad buttons and axes

You see 8 axes, 2 for left stick, 2 for right stick, and 4 for left D-PAD.

What is the expected behavior?
9 buttons should be reported. The controller has 2 shoulder, 2 triggers, axby buttons, and 1 home button. 

I'm not sure if left/right stick are clickable, but that would be two more buttons according to the spec.

What went wrong?
No buttons are reported.  This makes using this controller for games very difficult.

Did this work before? No 

Chrome version: 54.0.2830.0  Channel: n/a
OS Version: OS X 10.11.5
Flash Version: Shockwave Flash 23.0 r0
 

Comment 1 by a...@figma.com, Aug 17 2016

See the attached PNG.  Then controller id is:
  Nimbus (Vendor: 0111 Product: 1420)

Note that Firefox reports the axby and shoulder buttons, and stick axes, but not the left dpad.  The id numbers are different in Firefox.  

This whole API really needs to tie the buttons and axes to constants in a map, rather than relying on array order.  Joysticks tend to be a sparse map of buttons - this one doesn't have select or start, but has a home button.  There also really needs to be some way to set feedback on the joystick in terms of rumble.
Okay, I have to ask why build any Javascript API with so little longevity.   There are trivial changes to the API that would make code that relies on it much more robust to new controllers.

1. add a field to button with the code of the button.  You can still return an array of buttons then, but then I don't have to rely on a totally hardcoded array ordering.  I want to see BUTTON_A, BUTTON_B, BUTTON_X, BUTTON_Y.  You have these constants down in the Gamepad handling code, so just bubble them up.  Controllers pretty much follow the xbox 1 form factor these days, even if many of them don't have all the buttons.

2. make the axes field use a struct as well.  I still want to know the code, pressed, and value.  Simply returning an array of values isn't sustainable and can't be made sparse.  What the controller only had a single stick like on VR controllers?  Are these array orderings randomly generated?  They have no logical correspondence, or even any correspondence across browsers.

In FF, 4 axes are reported

0/1 - Left h/v
2/3 - Right h/v  

In Chrome, 8 axes are reported

0/1 - Left h/v
2 - Right h 
3 - dpad up
4 - dpad right   
5 - Right v    huh, that doesn't match std setup 
6 - dpad down
7 - dpad left 

Comment 3 by tkent@chromium.org, Aug 17 2016

Components: Blink>GamepadAPI

Comment 4 by kbr@chromium.org, Aug 20 2016

Cc: scottmg@chromium.org
Owner: bajones@chromium.org
Status: Assigned (was: Unconfirmed)
Brandon, may I assign this to you so it gets some attention?

I'll probably have to try and get my hands on one of these to figure out why the buttons aren't reporting. From the looks of it, this gamepad may be designed for Apple's new iOS-centric gamepad API, which may account in part for it's behavior. The same APi exists on OSX, but I wasn't aware that any devices actually used it yet. In any case, it doesn't seem to be reporting buttons the "normal" way (which among gamepads is sadly not all that normal).

As for why the API is structured the way that it is, that actually mirrors the underlying OS APIs pretty well. We don't get reports that "The X button was pressed", we get reports that button 2 was pressed. We don't see that the right stick was moved along the Y axis, we see that Axis 3 increased. APIs like XInput *do* actually provide a nicely labeled mapping, but they only support one very narrow range of devices.

What does that mean? Unless we, the developers, actually have the device in hand we have no idea what inputs map to what on your controller. In fact, we can't even make assumptions about the shape of your controller. Does it look like an XBox/PS4 pad? A NES controller? A steering wheel? A joystick? VR controller? We simply don't know. When we do, we add it to the standard mapping to try and make life easier for developers, but it's impractical to do that for every gamepad/joystick/controller-thing on the market.

Comment 6 by a...@figma.com, Aug 22 2016

The SteelSeries Nimbus controller is supposedly a standard OSX gamepad conformating to the GameController API and Apple MFi standard (only adopted on OSX/iOS). This was supposed to standardize the buttons similar to XInput.  

https://developer.apple.com/library/mac/documentation/ServicesDiscovery/Conceptual/GameControllerPG/IncorporatingControllersintoYourDesign/IncorporatingControllersintoYourDesign.html#//apple_ref/doc/uid/TP40013276-CH4-SW5

Mac Developer LibraryDeveloperSearch
Game Controller Programming Guide
 Table of Contents
Introduction
Incorporating Controllers into Your Game
Supplying Inputs to Your Game
Understanding the Controllers Supported by Apple
Requirements for Games That Support Game Controllers
Mapping Game Actions to Controls
Use Control Layouts That Players Are Familiar With
Identifying Controls Within Your Game
Building a Great User Experience with Controllers
Discovering and Connecting to Controllers
Working with Controller Elements
Controlling Input on tvOS
Appendix A: Checklist for Adding Controllers
Revision History
NextPrevious
Incorporating Controllers into Your Game

Game controllers provide new ways for players to interact with your game. When you design your game, it is up to you to decide whether a controller is appropriate and, if so, how it is used within your game. When a player has a controller, you might make it the exclusive source of input data to your game or it may be only one of many sources of input data. In the latter case, its controls are used along with data from other sensors, such as the touchscreen and motion sensors, to create an experience that builds upon the individual strengths of each input type. Therefore, to create a great user experience in your game, you should understand controllers, the kinds of input data they provide, and how a player uses the controls. Only then can you see how the controller can best be used in your game.

In this chapter, you’ll learn:

How the controls on a game controller compare to other kinds of input
The characteristics of game controllers supported by the Made-for-iPhone/iPod/iPad (MFi) program
Apple’s requirements for games that implement game controller support
How to map your game’s actions to controls on the controller
How to refer to specific controls on the controller
Specific actions you can take to ensure that game controllers add to your game’s experiences
Supplying Inputs to Your Game

Interactivity is a critical and necessary part of gameplay. Players continuously interpret sensory information (graphics, audio) provided by the game. From this data, players choose which actions to take. Then, players provide input data to the game to enact their plans. This decision loop (observe, orient, decide, act) happens continuously until the game completes. In an action game, this decision loop happens very quickly because the game does not pause while players are deciding how to act. In other games, where time does not pass automatically, players may be able to act more deliberately. But in all games, players decide what to do and then try to do it.

When you design a game, it’s a good idea to assess the different ways in which a player might control your game and then choose input methods that are appropriate. A game’s controls should feel natural to players. Although a player ought to feel challenged by your game, he or she should not feel challenged or frustrated by your game’s controls. Game controllers add to the input types available to you. Besides controllers, there are other sources of input data available to your game. For example:

Touch interface. The screen of an iOS device receives touch inputs from a player’s fingers. Touch input is a natural way for players to manipulate things onscreen but can often be imprecise; a player can select a somewhat small area quickly, but cannot select a pixel-sized area accurately. Also, because there is no tactile feedback provided for the objects being touched, it is difficult to manipulate objects that aren’t being viewed directly. For example, in the past, iOS games have simulated thumbsticks by drawing a picture of a thumbstick onscreen. These virtual controls do not work as well as an actual thumbstick because a player cannot pay close attention simultaneously to the thumbstick and the other elements onscreen.

Motion. The player manipulates the entire device, and the accelerometer and gyro systems provide input data to your game. As with the touch interface, motion controls can be very intuitive. But because the screen moves with the device, the onscreen images are constantly in motion.

Mouse. A player manipulates a mouse or trackpad, causing a cursor to move. Alternatively, the mouse movement can be interpreted as deltas and used to rotate or move a camera in the game. Even though mouse-based interfaces are indirect, they can provide pixel-level precision.

Keyboard. A keyboard provides discrete key inputs that can be mapped directly to actions in a game. Keyboards provide distinct control areas and tactile feedback.

The game controllers that Apple supports provide two additional options:

Analog buttons and triggers. Buttons and triggers provide tactile controls that can be mapped to specific game actions. When a player presses a button, an action is triggered. All buttons and triggers on MFi game controllers are pressure sensitive. After a certain amount of pressure has been applied to activate the button, the button measures the amount of force that is applied to the button. Thus, your game can either determine whether a button is pressed (discrete controls) or determine how much pressure has been applied to the button (analog).

Analog thumbsticks and analog directional pads. Thumbsticks and directional pads provide two axes of motion relative to a neutral, default position. As with buttons, thumbsticks and directional pads are analog controls, measuring how far the control has been moved from the neutral position. When the player releases a thumbstick or directional pad, it snaps back to this neutral position. Any dead-zone calculations are automatically performed by the Game Controller framework, so your game does not need to do this work.

Understanding the Controllers Supported by Apple

Apple has created specifications for distinct kinds of MFi game controllers. Although specific controllers vary, many common characteristics must be implemented strictly according to the specification.

see:
"Controllers Have a Standard or an Extended Control Layout
All of the controllers use either a standard control layout or an extended control layout. The standard control layout provides the following controls:"

The layout and exact visual config isn't as important, but would be useful informations in a string to match up to a visual button representation that the user can pick from.   What is important is the standardization of buttons/ids similar to XInput.  If we can get the sticks and buttons to come through in a consistent manner across browsers, then that will make incorporating controller input much easier.

Anyways, I hope that you don't give up on this.  I can always switch to a wireless Xbox controller hooked up to my OSX machine.  Just thought putting the tables in the browser and simply handing those down is a better with constant browser updates, than embedding them in the applications.

I can't currently hardcode the mappings, and this controller reports completely different id.  What if I don't have a start/select button, but I do have home and home is button 16.  The sparse button map on most controllers throws off even the "standard" mapping that is the only one the gamepad API supports, so now nothing maps properly.
 




Comment 7 by a...@figma.com, Aug 22 2016

Wiimote and even the Oculus Touch controller still have left/right stick, buttons, a, b, etc.  If you lookup in the map, and a particular button is not there, then obviously you find a mapping that fits.  That seems better than string mapping the several hundred strings that you'd need for every possible game config.  Maybe my game is like Limbo, and I just need left stick, and A/B.  That sparse map of buttons/sticks should map to a lot of devices, and I don't even really need to look at the mapping string.
I've ordered one of these controllers, primarily because it's the first GameController API-compatible device I've heard of that works on OSX. I'd like to get a code path for that into Chrome because it should allow for us to report any devices using the same API as a "standard" mapping. (I had looked into it in the past, but at the time all compatible controllers were iOS-only.) No promises as to when I'll be able to get support in, but at least I'll have the device for testing.

Please understand, though, that the reason the Gamepad API is structured this way is to try and provide support for precisely this type of device: One which we haven't seen before but may report something usable through standard HID channels. Seems like that route failed here, but it provides *something* for a great number of devices that we'd never have anticipated otherwise. we could certainly limit the API to only devices which we have been able to successfully map to a standard mapping, but then your only option would be to buy one of a few "Chrome approved" gamepads, which would be a shame. 

Comment 9 by a...@figma.com, Aug 22 2016

That's not the choice that I'm suggesting here. I'm saying expose the mapping that you have internally of well defined buttons/axes to the caller.  As the browser supports more and more controllers, the browser can update it's mapping.  You can still have an "unknown" mapping which doesn't supply the mapping, and callers can still hardcode for those, but I doubt they will unless it's a short-term demo.

Now we can lookup from a standard set of buttons, haptics, and axes (a/b/x/y, leftAxisX, leftAxisY, leftUpperMotor) to get the array index on that gamepad.  I can play Limbo with a nunchuck Wiimote, or a gameboy-like gamepad, or an Oculus Touch.  The buttons and axes and haptics should be similar.  If I need all 15 buttons, I can even check that the ones that I need are in the map, without having to string match to various "standard" mapping types.

I would have originally said to move the axes and buttons to structs that include these gamepad neutral codes, but this sparse map seems like the simpler option.

Now on the Nimbus which doesn't have select/start buttons, you have all of the standard buttons except those in this mapping.  But I don't need select/start.  It can still be the compacted array of buttons, axes, and haptics.  It's just that now I can go from a pad neutral value to an array index.  

FYI: I got my Nimbus and was able to pull together a gamepad implementation using Apple's GameController API the same night. CL is here: https://codereview.chromium.org/2271353002/

I think the point of disconnect that we have here is when you say "expose the mapping that you have internally of well defined buttons/axes to the caller". *Any* time we have a mapping like that available, we expose it as a "standard" mapped device. (The CL I linked above does exactly that for GameController devices.) The issue is that a large majority of the time we don't have any such mapping provided by the system.

Once a gamepad is mapped to the standard layout, whether you're given an array of inputs or a sparse map is primarily syntactic sugar. It's simple to add some constants to your JS that let you refer to the inputs with a more logical name if those types of devices are all you care about:

var Buttons {
  A: 0,
  B: 1,
  X: 2,
  Y: 3,
  // ETC
};

var Axis {
  LeftX: 0,
  LeftY: 1,
  RightX: 2,
  RightY: 3
};

var aValue = gamepad.buttons[Button.A];
var leftStickXValue = gamepad.axes[Axis.LeftX];

Comment 11 by a...@figma.com, Aug 24 2016

That's still not going to work for the extensibility in the short term.  The Nimbus has all of the standard buttons except select/start.  So now are you going to return a different "standard2" mapping?   I'd be fine with that being reported as a standard mapping, but I still need to know what buttons are available.  I've already put these constants in my code, and they didn't line up between Firefox and Chrome on anything.

There are still more values that are needed for each axis (range: 0 to 1, -1 to 1, deadzones, etc).  On buttons is it discrete or not, touched, etc.  I'm suggesting that really axes and buttons should be similar structs with the id, but I thought the map would be the simpler approach to add where for now you just get the button to array index.

A:0, B:1  <- exactly what you have in the diagram for standard mapping, but 14 and 15 (guessing for the ids) won't be in it for the Nimbus, and Home will be 14.  It's mostly a standard mapping.  This will save you a and me a lot of work on the handling side where I only need a few buttons, or can adapt to missing buttons.

Code stops depending on the specific array index of the "standard" mapping, and uses the constants that you set for A/B/X/Y.

Comment 12 by a...@figma.com, Aug 24 2016

I've written for Xbox, DirectInput, XInput, and a lot of controller setups.  I know that they're somewhat randomly designated id's if there's not standardization, but GameController API and XInput have at least started that process.  You can always specify unknown if you don't know how to map a gamepad and leave it to the caller.  Empty mapping table, but arrays that are non-zero.

Comment 13 by a...@figma.com, Aug 24 2016

for mapping="standard"

buttonMap={A:0,B:1,X:2,Z:3, ...}
axisMap={LeftStickX:0, LeftStickY:1, ...}
hapticMap={LeftMotor:0, ...}

or one map
map={LeftMotor:0, A:0, LeftStickX:0, ...}

Nimbus can't have Home button be 16, since it's missing select/start, but it's still mostly a standard layout.  I can query the map above for select/start and find out that it's not there.  Also the browser has remapped HOME:14 instead of HOME:16, so I dno't have to guess of yet another "unknown" mapping string.  Most OSX MFi standard controllers will have a sparse set of the "standard" mapping.

As I suggested earlier, even the Wiimote and Oculus Touch map to many of these buttons.  But again it's either a sparse or superset of the standard buttons/axes.  I haven't seen you mention what happens in the API when you have a sparse/superset.  Currently, you just get that the joystick is "unknown" and then that's just foisting the work of making these tables on every app that wanted a few lines to deal with a gamepads (and not fully generally HID).  So now I'm faced with updating Unity-like tables for every controller in my tiny web demo?




Comment 14 Deleted

Comment 15 by a...@figma.com, Aug 24 2016

So how do I know if the buttons are there and not pressed, or just not there?
Status: Fixed (was: Assigned)
Project Member

Comment 17 by bugdroid1@chromium.org, Aug 25 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3b3379eceba787068c53d37c0f1466f2c271d336

commit 3b3379eceba787068c53d37c0f1466f2c271d336
Author: bajones <bajones@chromium.org>
Date: Thu Aug 25 00:05:24 2016

Added support for GameController devices on OSX

This is Apple's new gamepad API for Mac and iOS devices, which imposes a ridgid
set of guidelines on controller capabilities, similar to XInput on Windows. This
allows us to provide a "standard" mapping for every compatible controller. While
there aren't many on the market right now, it should make us more future proof
and provide better compatibility with newer controllers.

BUG= 638588 

Review-Url: https://codereview.chromium.org/2271353002
Cr-Commit-Position: refs/heads/master@{#414211}

[modify] https://crrev.com/3b3379eceba787068c53d37c0f1466f2c271d336/device/gamepad/BUILD.gn
[add] https://crrev.com/3b3379eceba787068c53d37c0f1466f2c271d336/device/gamepad/game_controller_data_fetcher_mac.h
[add] https://crrev.com/3b3379eceba787068c53d37c0f1466f2c271d336/device/gamepad/game_controller_data_fetcher_mac.mm
[modify] https://crrev.com/3b3379eceba787068c53d37c0f1466f2c271d336/device/gamepad/gamepad_pad_state_provider.h
[modify] https://crrev.com/3b3379eceba787068c53d37c0f1466f2c271d336/device/gamepad/gamepad_platform_data_fetcher.h

Comment 18 by a...@figma.com, Aug 25 2016

Thanks Brandon for your dialog on this.  I think that MFi standard will help a lot.  Especially now that AppleTV supports apps and controllers.   I'm looking at VR on iOS, and didn't want to write a native app.  The Gamepad API is almost there, and the realization that I had was that the missing buttons represented a gap in the "standard" array.  Even if you just return 0's in the array for missing buttons, that kind of works, but it's still not possible to know if those would ever not be zero (buttons missing or just not pressed).  I could see detecting the missing buttons, and adapting controls to different buttons/axes.

Comment 19 by a...@figma.com, Aug 25 2016

Also GameController support on Chrome iOS would also allow me to do WebVR much easier.  That's why I was using the MFi controller in the first place.  It's nice not to have to use the Xbox Adapter.   The only alternative for Cardboard right now is using a web-socket to a laptop that.  
It's worth noting that this update will have no effect on Chrome on iOS. Exposing things like the HTML5 Gamepad API is completely under Apple's control and we have no influence over it. Sorry.

Comment 21 by a...@figma.com, Aug 28 2016

This .mm file is missing the menu button (standard button index 16).  That seems like a pretty important button to MFi (like the Xbox menu button).  

There's still the missing array index and/or functionality to identify missing from non-pressed buttons, but oh well.  At least this is better than no buttons.
I noticed that wasn't mapped but I assumed it was because the OS stole it like the Xbox button on an Xbox. It looks like it's not available via https://developer.apple.com/reference/gamecontroller/gcextendedgamepad -- is there some other way to retrieve it?
For the record that button is sorta exposed by the API, but it's not treated like a normal button. The GameController API refers to it as the "Pause" button, and it's the only button that requires you to listen to an event in order to capture it. (See https://developer.apple.com/library/mac/documentation/GameController/Reference/GCController_Ref/index.html#//apple_ref/occ/instp/GCController/controllerPausedHandler) As a result it works differently than the other buttons. I know when it's pressed but not when it's released (or maybe it's visa-versa?) That means that the button will show up as a quick little "blip!" of data that may be missed entirely due to the nature of our gamepad polling architecture. This makes the button's behavior confusing and unreliable, and I'd rather not expose something that flaky.

We could treat it as a toggle. One press locks it on, the next off, but no other inputs I'm aware of work like that to date.
Brandon you are fast. I just tried out the support, and it's working great.  Only issue is that http://html5gamepad.com/ is reporting two gamepads (the broken one) and the new "standard" one.  Could just be a bug in that implementation.

Also thumbstick deadzones are automatically applied by the GameController API, so at some point you may want to report 0 for the deadzone on the axes.  Buttons ramp up on pressure before being pressed though.  There's no api exposed for this yet, but min/max and deadzone are common values.
Not sure that I can do much about the deadzones, since as you noted they're built into the API. It should already be reporting zero any time the stick is in the deadzone, are you seeing different behavior?

As for the two sticks showing up, known issue that I'm actively researching. It's not clear to me that there's a way to associate the GameController device with a IOHIDDeviceRef. (See http://stackoverflow.com/questions/33509296/supporting-both-gccontroller-and-iohiddeviceref). Once I've established a reliable way to associate them I'll filter out the devices from the HID pass. In the meantime, the GameController variant should always be the first one that shows up.
Also the vertical stick axes seem reversed.  1 would be better at the top, and -1 at the bottom.  The top controller seems to report this in html5gamepad.com, but the same bottom controller is reversed, and my own test app seems flipped too.  

I'm saying that at some point, it may be important to report the XInput and GameController stick deadzone. XInput was always an integer deadzone, but that could be normalized. 
I don't think you need to negate these:

    pad.axes[AXIS_INDEX_LEFT_STICK_Y] =
        -[[[extended_gamepad leftThumbstick] yAxis] value];
    pad.axes[AXIS_INDEX_RIGHT_STICK_Y] =
        -[[[extended_gamepad rightThumbstick] yAxis] value];
Gamepad spec defines the "standard" layout as having Y axes that are negative up, positive down. See: https://w3c.github.io/gamepad/#remapping

I don't know what drove that decision, I wasn't around when it was made.
I see, but can't recall the typical standard there.  Flipped those, and fixed some of the ids that I was using now that everything is mapped to the standard id.  This is fun now.  Thanks again.
You might already have these, but these are the integer deadzones for XInput and the 360 controller.  It would be best not to expose these if the api could handle these internally, but I was suggesting that if not, you could return 0 for the stick deadzone on the GameController API.   

#define XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE  7849
#define XINPUT_GAMEPAD_RIGHT_THUMB_DEADZONE 8689
#define XINPUT_GAMEPAD_TRIGGER_THRESHOLD    30

Sign in to add a comment