New issue
Advanced search Search tips

Issue 845011 link

Starred by 4 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug

Blocking:
issue 843511



Sign in to add a comment

various crash handling tests fail under win/asan

Project Member Reported by thakis@chromium.org, May 20 2018

Issue description

I added crashpad_test to the win/asan bots recently. Some of the tests don't pass:

https://ci.chromium.org/buildbot/chromium.clang/CrWinAsan/623

CrashpadClient.HandlerLaunchFailureCrash
CrashpadClient.HandlerLaunchFailureDumpAndCrash
CrashpadHandler.ExtensibilityCalloutsWork
SimulateCrash.ChildDumpWithoutCrashing
ExceptionSnapshotWinTest.ChildCrash



[ RUN      ] CrashpadClient.HandlerLaunchFailureCrash
=================================================================
==2724==ERROR: AddressSanitizer: breakpoint on unknown address 0x00000000 (pc 0x0025e0f3 bp 0x0019f178 sp 0x0019f0c0 T0)
==2724==The signal is caused by a READ memory access.
==2724==Hint: address points to the zero page.
[0520/062249.577:ERROR:crashpad_client_win.cc(491)] CreateProcess: The system cannot find the file specified. (0x2)
    #0 0x25e0f2 in crashpad::test::`anonymous namespace'::HandlerLaunchFailureCrash::WinMultiprocessChild+0x82 (e:\b\s\w\ir\out\Release\crashpad_tests.exe+0x4ae0f2)
    #1 0x25e022 in crashpad::test::WinMultiprocess::ChildProcessHelper<crashpad::test::(anonymous namespace)::HandlerLaunchFailureCrash>::Run C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\win\win_multiprocess.h:152
    #2 0x25defe in crashpad::test::`anonymous namespace'::CrashpadClient_HandlerLaunchFailureCrash_Test::TestBody C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\client\crashpad_client_win_test.cc:131
    #3 0x627379 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2402
    #4 0x5fda19 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,void> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472
    #5 0x5fd588 in testing::Test::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2491
    #6 0x5ffb32 in testing::TestInfo::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2667
    #7 0x600ddf in testing::TestCase::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2785
    #8 0x61b59d in testing::internal::UnitTestImpl::RunAllTests C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:5047
    #9 0x627e29 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2402
    #10 0x61ad29 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472
    #11 0x61a9d0 in testing::UnitTest::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4663
    #12 0x8fe4b4 in main C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\gtest_main.cc:95
    #13 0xa6dceb in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283
    #14 0x75393369 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73369)
    #15 0x779f9901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901)
    #16 0x779f98d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: breakpoint C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\client\crashpad_client_win_test.cc:123 in crashpad::test::`anonymous namespace'::HandlerLaunchFailureCrash::WinMultiprocessChild
==2724==ABORTING
../../third_party/crashpad/crashpad\test/win/win_multiprocess.h(68): error: Expected equality of these values:
  exit_code
    Which is: 1
  parent_process.exit_code_
    Which is: 4294930433
Stack trace:
Backtrace:
	StackTraceGetter::CurrentStackTrace [0x005C894A+330] (C:\b\c\b\CrWinAsan\src\third_party\googletest\custom\gtest\internal\custom\stack_trace_getter.cc:22)
	testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x005EDDDB+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:810)
	testing::internal::AssertHelper::operator= [0x005ECCF9+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:386)
	crashpad::test::`anonymous namespace'::CrashpadClient_HandlerLaunchFailureCrash_Test::TestBody [0x0025DA9C+2364]
[  FAILED  ] CrashpadClient.HandlerLaunchFailureCrash (172 ms)




[ RUN      ] CrashpadClient.HandlerLaunchFailureDumpAndCrash
=================================================================
==4148==ERROR: AddressSanitizer: access-violation on unknown address 0x00000004 (pc 0x005ed75c bp 0x016bf9e4 sp 0x016bf81c T0)
==4148==The signal is caused by a READ memory access.
==4148==Hint: address points to the zero page.
    #0 0x5ed75b in testing::UnitTest::AddTestPartResult C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4538
    #1 0x5fc47f in testing::internal::ReportFailureInUnknownLocation C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2269
    #2 0x627eae in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2410
    #3 0x61ad29 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472
    #4 0x61a9d0 in testing::UnitTest::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4663
    #5 0x8fe4b4 in main C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\gtest_main.cc:95
    #6 0xa6dceb in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283
    #7 0x75393369 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73369)
    #8 0x779f9901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901)
    #9 0x779f98d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: access-violation C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4538 in testing::UnitTest::AddTestPartResult
==4148==ABORTING
../../third_party/crashpad/crashpad\test/win/win_multiprocess.h(68): error: Expected equality of these values:
  exit_code
    Which is: 1
  parent_process.exit_code_
    Which is: 4294930433
Stack trace:
Backtrace:
	StackTraceGetter::CurrentStackTrace [0x005C894A+330] (C:\b\c\b\CrWinAsan\src\third_party\googletest\custom\gtest\internal\custom\stack_trace_getter.cc:22)
	testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x005EDDDB+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:810)
	testing::internal::AssertHelper::operator= [0x005ECCF9+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:386)
	crashpad::test::`anonymous namespace'::CrashpadClient_HandlerLaunchFailureDumpAndCrash_Test::TestBody [0x0025F1BC+2364] (C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\client\crashpad_client_win_test.cc:155)
[  FAILED  ] CrashpadClient.HandlerLaunchFailureDumpAndCrash (109 ms)



[ RUN      ] CrashpadHandler.ExtensibilityCalloutsWork
=================================================================
==4600==ERROR: AddressSanitizer: unknown exception on unknown address 0x00000000 (pc 0x00000000 bp 0x0162ebf8 sp 0x0162e4fc T0)
==4600==Hint: pc points to the zero page.
==4600==The signal is caused by a READ memory access.
==4600==Hint: address points to the zero page.
    #0 0x26d7f8 in crashpad::test::`anonymous namespace'::CrashWithExtendedHandler::WinMultiprocessChild+0x5e8 (e:\b\s\w\ir\out\Release\crashpad_tests.exe+0x4bd7f8)
    #1 0x26d09b in crashpad::test::WinMultiprocess::ChildProcessHelper<crashpad::test::(anonymous namespace)::CrashWithExtendedHandler>::Run C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\win\win_multiprocess.h:152
    #2 0x26cf66 in crashpad::test::`anonymous namespace'::CrashpadHandler_ExtensibilityCalloutsWork_Test::TestBody C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\handler\crashpad_handler_test.cc:138
    #3 0x627379 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2402
    #4 0x5fda19 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,void> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472
    #5 0x5fd588 in testing::Test::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2491
    #6 0x5ffb32 in testing::TestInfo::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2667
    #7 0x600ddf in testing::TestCase::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2785
    #8 0x61b59d in testing::internal::UnitTestImpl::RunAllTests C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:5047
    #9 0x627e29 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2402
    #10 0x61ad29 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472
    #11 0x61a9d0 in testing::UnitTest::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4663
    #12 0x8fe4b4 in main C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\gtest_main.cc:95
    #13 0xa6dceb in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283
    #14 0x75393369 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73369)
    #15 0x779f9901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901)
    #16 0x779f98d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: unknown exception C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\handler\crashpad_handler_test.cc:76 in crashpad::test::`anonymous namespace'::CrashWithExtendedHandler::WinMultiprocessChild
==4600==ABORTING
../../third_party/crashpad/crashpad\test/win/win_multiprocess.h(68): error: Expected equality of these values:
  exit_code
    Which is: 1
  parent_process.exit_code_
    Which is: 485163226
Stack trace:
Backtrace:
	StackTraceGetter::CurrentStackTrace [0x005C894A+330] (C:\b\c\b\CrWinAsan\src\third_party\googletest\custom\gtest\internal\custom\stack_trace_getter.cc:22)
	testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x005EDDDB+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:810)
	testing::internal::AssertHelper::operator= [0x005ECCF9+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:386)
	crashpad::test::`anonymous namespace'::CrashpadHandler_ExtensibilityCalloutsWork_Test::TestBody [0x0026CA71+2385]
[  FAILED  ] CrashpadHandler.ExtensibilityCalloutsWork (140 ms)



[ RUN      ] SimulateCrash.ChildDumpWithoutCrashing
../../third_party/crashpad/crashpad/snapshot/win/exception_snapshot_win_test.cc(211): error: Expected: (snapshot.Exception()->Context()->InstructionPointer()) < (dump_near_ + kAllowedOffset), actual: 15275245 vs 15275188
Stack trace:
Backtrace:
	StackTraceGetter::CurrentStackTrace [0x005C894A+330] (C:\b\c\b\CrWinAsan\src\third_party\googletest\custom\gtest\internal\custom\stack_trace_getter.cc:22)
	testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x005EDDDB+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:810)
	testing::internal::AssertHelper::operator= [0x005ECCF9+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:386)
	crashpad::test::`anonymous namespace'::SimulateDelegate::ExceptionHandlerServerException [0x003C5B47+1959]
	crashpad::ExceptionHandlerServer::OnNonCrashDumpEvent [0x008122D6+374] (C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\util\win\exception_handler_server.cc:565)
	RtlRegisterWait [0x77A31CF2+788]
	RtlIsCriticalSectionLockedByThread [0x77A10CE8+766]
	TpCallbackIndependent [0x77A10951+1808]
	BaseThreadInitThunk [0x7539336A+18]
	RtlInitializeExceptionChain [0x779F9902+99]
	RtlInitializeExceptionChain [0x779F98D5+54]
[  FAILED  ] SimulateCrash.ChildDumpWithoutCrashing (94 ms)



[ RUN      ] ExceptionSnapshotWinTest.ChildCrash
=================================================================
==4632==ERROR: AddressSanitizer: breakpoint on unknown address 0x00000000 (pc 0x001a148b bp 0x0091f9c4 sp 0x0091f240 T0)
==4632==The signal is caused by a READ memory access.
==4632==Hint: address points to the zero page.
    #0 0x1a148a in wmain C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\snapshot\win\crashpad_snapshot_test_crashing_child.cc:49
    #1 0x20535b in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283
    #2 0x75393369 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73369)
    #3 0x779f9901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901)
    #4 0x779f98d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: breakpoint C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\snapshot\win\crashpad_snapshot_test_crashing_child.cc:49 in wmain
==4632==ABORTING



Probably should disable these tests under asan for now and then investigate later.
 

Comment 1 by thakis@chromium.org, May 20 2018

Blocking: 843511
Owner: scottmg@chromium.org
Status: Started (was: Untriaged)
I just looked into these tests. These four

CrashpadClient.HandlerLaunchFailureCrash
CrashpadClient.HandlerLaunchFailureDumpAndCrash
CrashpadHandler.ExtensibilityCalloutsWork
ExceptionSnapshotWinTest.ChildCrash

are failing because ASAN captures an expected crash without letting crashpad see it, and either terminates the process with an unexpected error code, or causes a hang because the test fixture doesn't notified. So I think these are fine to leave disabled when running under ASAN.

Unless there's a way to tag specific crash locations as uninteresting so that ASAN will let those through and we can check the rest of the code in those tests?

SimulateCrash.ChildDumpWithoutCrashing is doing an instruction pointer location compare, and the ASAN instrumentation made the code larger than expected. I posted a CL to bump up the value when under ASAN. https://chromium-review.googlesource.com/c/crashpad/crashpad/+/1075715 I also upstreamed the rest of the disabling CL https://chromium-review.googlesource.com/1067151 there.
Project Member

Comment 4 by bugdroid1@chromium.org, May 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/crashpad/crashpad.git/+/5191992ae54c5e349130e0e9394349568ee11b8b

commit 5191992ae54c5e349130e0e9394349568ee11b8b
Author: Scott Graham <scottmg@chromium.org>
Date: Tue May 29 17:04:18 2018

win: Fix SimulateCrash.ChildDumpWithoutCrashing under ASAN, disable others

SimulateCrash.ChildDumpWithoutCrashing needed a larger threshold due to
ASAN instrumentation.

These tests expect children to crash, but ASAN captures the exception
before letting Crashpad handle it:

CrashpadClient.HandlerLaunchFailureCrash
CrashpadClient.HandlerLaunchFailureDumpAndCrash
CrashpadHandler.ExtensibilityCalloutsWork
ExceptionSnapshotWinTest.ChildCrash

(which is an upstreaming of https://chromium-review.googlesource.com/1067151).

Additionally, because Chrome doesn't build all, I noticed a missing
dependency on a test binary which is added here.


Bug: chromium:845011
Change-Id: I5c3ae5673512be29edad21e7d20dd57b8b5ce2bf
Reviewed-on: https://chromium-review.googlesource.com/1075715
Reviewed-by: Joshua Peraza <jperaza@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>

[modify] https://crrev.com/5191992ae54c5e349130e0e9394349568ee11b8b/client/crashpad_client_win_test.cc
[modify] https://crrev.com/5191992ae54c5e349130e0e9394349568ee11b8b/handler/BUILD.gn
[modify] https://crrev.com/5191992ae54c5e349130e0e9394349568ee11b8b/handler/crashpad_handler_test.cc
[modify] https://crrev.com/5191992ae54c5e349130e0e9394349568ee11b8b/snapshot/win/exception_snapshot_win_test.cc

Comment 5 by thakis@chromium.org, May 29 2018

rnk: Do you have an opinion on comment 3?

Do we disable "child crashes" tests in chromium's tests under asan too?

Comment 6 by r...@chromium.org, May 29 2018

Cc: vitalyb...@chromium.org
Maybe the easiest way forward would be to disable ASan's crash handler in all crashpad_tests.

Vitaly, I remember ASan recently changed all the defaults around allow_user_segv_handler/handle_segv, etc, because it was a problem for Go or some other customer. I remember that Clusterfuzz wants all fuzz targets to use the same crash report format for ASan error reports and regular crashes, which are typically null pointer derefs. Should we try to handle access violation exceptions by default on Windows? Should clusterfuzz set their own custom ASAN_OPTIONS to get the behavior they want?

If we go that way, ASan will violate fewer application expectations, and most crash handler tests should start to pass without modification.

In the short term, disabling tests seems fine.
Yes, but not on Windows: http://llvm-cs.pcc.me.uk/projects/compiler-rt/lib/sanitizer_common/sanitizer_win.cc#852

Default is to set Asan as signal handler, but let application to override it.
Clusterfuzz wants Asan as the only signal handler, blocking overrides from application. Clusterfuzz sets behavior with ASAN_OPTIONS.
But these should be irrelevant to Windows tests?

Comment 8 by r...@chromium.org, Jun 14 2018

Cc: thakis@chromium.org etienneb@chromium.org mark@chromium.org
 Issue 782907  has been merged into this issue.

Comment 9 by r...@chromium.org, Jun 20 2018

Owner: r...@chromium.org
Status: Assigned (was: Started)
Summary: various crash handling tests fail under win/asan (was: various crashpad_test tests fail under win/asan)
I don't think we can do what ASan on Linux does, where ASan sets the unhandled exception filter, and then lets the user override it, because the CRT always overrides it. We inherently have to fight the CRT to install our "signal handler" if we want it to work at all.

I think we should add this code to src/build/sanitizers/sanitizer_options.cc:

#if defined(OS_WIN)
SANITIZER_HOOK_ATTRIBUTE const char *__asan_default_options() {
  // Disable ASan's access violation handler by default. It interferes too much
  // with crash testing. ClusterFuzz and other fuzzing tools should explicitly
  // override these in the ASAN_OPTIONS environment variable for uniform crash
  // reporting.
  return "handle_segv=0 handle_abort=0 handle_sigill=0 handle_sigfpe=0";
}
#endif // OS_WIN

That will cause ASan to install its exception filter, but always delegate to Chrome's (or crashpad's) exception filter by default.

I verified that ClusterFuzz always sets ASAN_OPTIONS=handle_segv=1 handle_sigill=1... in the environment, so its behavior won't be affected. ClusterFuzz will get ASan reports for both access violation crashes and ASan UAF reports.
Project Member

Comment 10 by bugdroid1@chromium.org, Jun 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d7685a5fd09bcf5144008cec5d8f3396dda6eacd

commit d7685a5fd09bcf5144008cec5d8f3396dda6eacd
Author: Reid Kleckner <rnk@google.com>
Date: Wed Jun 20 20:34:31 2018

Change headless crash handler test expectations under ASan

ASan on Windows changes the exit code to 1, which base categorizes as
the process being killed.

R=dgozman@chromium.org
BUG=845011

Change-Id: I42149965f2904a2a8349a00f4a2dd70351d2eee1
Reviewed-on: https://chromium-review.googlesource.com/1108448
Reviewed-by: Dmitry Gozman <dgozman@chromium.org>
Commit-Queue: Reid Kleckner <rnk@chromium.org>
Cr-Commit-Position: refs/heads/master@{#568994}
[modify] https://crrev.com/d7685a5fd09bcf5144008cec5d8f3396dda6eacd/headless/lib/headless_devtools_client_browsertest.cc

Sign in to add a comment